Gary Thompson FSTech program manager for canceled TxDOT toll systems contract resigned, TxDOT felt misled
2012-08-10: Federal Signal Technologies (FSTech) Gary A Thompson program manager for their canceled $68m toll back office contract with Texas DOT left the company August 2. Neither Thompson nor FSTech would comment at the time, but right or wrong, it appears he has been held responsible for a major contract gone bad. CORRECTION Aug22: Thompson resigned, he wasn't fired as we were initially informed and reported here. see http://www.tollroadsnews.com/node/6140
The contract was a big and prestigious one: formally "Agreement for the Provision of Statewide Toll Collection Service Center System and Operations and Central Texas Turnpike System Toll Operations," Contract No. 87-1RFPC007. It is abbreviated (POSTCSCSOCTTSTO) and is not to be confused with the contract for Toll Operations & Customer Service Center Systems and Operations (TOCSCSO) and an older, less comprehensive FSTech system called VTX.
TxDOT Toll Operations Division (TOD) say FSTech was not meeting schedule in delivering the new statewide back office toll system and they felt they had been misled by FSTech initially. They cancelled the contract "for convenience" July 17 after just seven months - a most unusual action in contracting at least with a unit of a large company like Federal Signal.
TxDOT/TOD thought they were buying from FSTech a back office system named Venture CSS the code of which was written and merely needed tweaking to meet their needs, as TxDOT tells it. But they came to suspect the system code was being written almost from scratch.
They felt deceived.
FSTech for its part claimed the RFP and contract contained 140 items that contradicted one another or that required clarification, changes and an extended schedule. They attempted to introduce component off-the-shelf software for auxiliary functions, deferred tests and argued about the contract schedule.
Under Texas Open Records law TxDOT has provided us close to 30 pieces of correspondence on the dispute - letters as .pdf and .doc files - between the Toll Operations Division (TOD) of TxDOT and FSTech, and FSTech and TOD. Most of them are letters between Gary A Thompson, FSTech program manager in Austin, and Kathy Garrett, IT Project Coordinator in Austin. Others are copied on the letters.
Garrett was the contract manager for the Toll Operations Division (TOD) TxDOT
FSTech's proposal to TxDOT/TOD made in competition with two others (TransCore and ACS/Xerox) in the spring of 2011 said that their system "currently meets 97% of these (RFP) requirements." TxDOT weighed this 97% claim heavily in awarding the contract to FSTech, we understand, considering FSTech the low risk proposer.
FSTech were given the official Notice to Proceed - 'the contract is on' - December 9, 2011.
Relations between TOD and FSTech seem to have gone bad as early as February this year. A letter February 29 from Thompson to Kathy Garrett is stiffly worded. It claims and later lists over 9 pages 140 items in an Exhibit I of the executed contract that it says contradict or make unclear the work required.
The letter asked for instructions on the 140 items within two weeks - by March 12. Then it suggests TxDOT may delay the project.
"We absolutely don't have the luxury of waiting very long before this will imp[act our delivery schedule... For every day these items are not addressed by TOD there is a potential for a day-for-day slippage of our delivery schedule. This is absolutely one of our most critical path tasks that we face as we work towards the goal of delivering the Venture CSS software by September 10, 2012 (deadline)," writes FSTech program manager Thompson.
140 contradictions cited by FSTech
In response TOD on March 6 said it would have the responses to the 140 items by March 12 but sought reassurance that the software was "97% ready." TOD was apparently beginning to worry that FSTech was developing the system software rather than doing tweaks to an off-the-shelf system they had ready.
Responses on the 140 items were delivered in a 9-page letter March 12 by TOD to FSTech. The covering letter does complain that a number of the items did not pose any question or contain any comment so TOD could not respond.
There was then a sharp dispute over FSTech's request to TOD to accept a subcontractor Gila LLC dba Municipal Services Bureau (MSB) to do TxTag transponder fulfillment and inventory management using a program called Business Ink (BI.)
TOD flatly rejected this April 6 this as contrary to the FSTech proposal and contract under which those functions would be handled by FSTech's new Venture CSS software.
FSTech's Thompson wrote April 11 that TOD had unreasonably dismissed the request to have MSB as a subcontractor and that use of Business Ink (BI) software would only enhance their Venture CSS' ability to track and report the status of TxTags.
April 16 Kathy Garrett for TOD responds that the use of BI is unacceptable and contrary to the terms of FSTech's proposal and contract. It would require regularly updating the Venture CSS from an auxiliary system and would add "an additional layer of complexity to the (TxTag) fulfillment process."
TxDOT in mid April read Thompson's strong push to use MSB's BI software as panic based on the lack of any such capability in Venture CSS as promised.
Cure letters start, legal counsel attend meetings
April 17 TOD wrote FSTech laying out ten "deficiencies" in the Venture software customer website, asking they be addressed along with a new plan no later than April 24.
Factory Acceptance Tests (FATs) were cancelled by FSTech the next day, and again two days later. Another four days later FSTech wrote saying they needed to suspend the testing to establish what they "technically need to do to be successful."
The same day, April 23 FSTech's Gary Thompson proposed using subcontractor Faneuil's Service Command software.
The next day Thompson said they needed three weeks (to mid-May) to produce a new acceptance plan on when tests of Venture CSS could resume.
In a bit of light relief TOD wrote May 23 that there was a complication with a 'Historically Underutilized Business' sub-contractor named TLC (Tender Loving Care?) Engineering Inc, a company that FSTech partner Faneuil rejected, which put FSTech out of compliance with the contract. There is no further correspondence on TLC Engineering or the federal government's strictures on "Historically Underutilized Businesses," so that must have been settled.... with TLC?
May 24 Thompson submitted the "mid-May" revised schedule.
May 25 TOD's Garrett wrote FSTech complaining that it is still proposing MSB software and systems rather than those promised as part of Venture CSS. The letter express disbelief ("this simply cannot be the case") about assurances about the state of Venture CSS having functions that were being held for later demonstration.
She threatens to "simply reject" anything not documented in future indicating a growing distrust. She says TOD's "overarching concern at this time is not exclusively the lack of quality control, but the fact that system functions required in the new Venture (CSS) system and specified in the technical provisions of the contract were in many cases simply unaddressed" in the plan.
Lawyers called in
TxDOT lawyers had been alerted to possible litigation and began attending the weekly 'status meetings' between FSTech and TOD May 24.
May 30 TOD's Garrett formally rejected the FSTech revised Venture CSS implementation schedule saying it did not provide time for full testing: "The substantial abbreviation of key testing proposed would not only prevent a full billing cycle of testing and insufficient time to satisfactorily test all elements of system functionality and integration, but is also contrary to explicit contractual terms."
Functionality demonstrated so far "falls well short of this expectation," Garrett wrote.
She said the request for modifications "at this late stage of the implementation is highly irregular." However as a demonstration of good faith they would allow "some revisions." Garrett said that they had to allow 45 days for POT (not marijuana we're told but Prototype On-site Tests.) They had to get cycles-stories-release documentation, factory acceptance test distribution and a "burn down chart" (we got it first as BUM down suggesting seat-of-the-pants or armchair assessment, because in lower case the r and n can't be distinguished from an m. But it is "BURN down" perhaps testing to destruction?)
Then came the threat to cancel: "Failure to comply with this request may result in a notice of default under the Agreement."
FSTech continued to be viewed "as a vital partner" in the success of TxTag, Garrett wrote, but: "It is essential that FST(ech) demonstrate ability to recover from the substantial standing deficit between the scheduled program delivery and performance demonstrated to date."
May 31 Thompson responded defensively to the May 25 TxDOT complaint detailing new quality controls he was implementing.
June 6 Thompson said that a review FSTech had done of the contract showed that Garrett had misunderstood its provisions. He asked for a face to face meeting to "explain why we believe certain of the scheduling demands TxDOT (TOD) has articulated are inconsistent with the terms of the contract and with the logic and natural sequence of events in the project..."
Thompson proposed leaving some testing until after commencement of customer service center operations.
And, he said, they calculated the contractual obligation for service commencement to be September 10, 2012.
POTs before pans
June 14 TxDOT rejected an FSTech proposal to defer POT (protoype on-site testing not parallel operations tests - see definitions below.) TxDOT felt FSTech was desperately casting around for excuses to avoid testing, we understand. Garrett said in her June 14 letter that the contract defined the purpose of POT as "to provide sufficient confidence to TxDOT that the CSC (customer service center) system is ready for final cutover."
She reminded Thompson of FSTech's Program Management Plan which has POT as Milestone 3 before startup which is Milestone 4. (see Table 1 nearby)
June 20 Garrett wrote Thompson again saying the Venture CSS tests held at FSTech offices June 15 were unsatisfactory. Only some of the scheduled tests were done and "many were found to be deficient."
She added: "The business team is concerned that FAT (final acceptance testing) testing continues to erode confidence that the system functionality is on target for this late point in the overall implementation process."
Pulling staff from operations, alleged
Around this time TOD complained that FSTech was pulling staff from their work on the earlier TOCSCSO/VTX contract to work on the Venture CSS contract causing VTX operations to suffer.
TxDOT filed a claim for $117,000 against FSTech for VTX problems caused by lack of attention, they said, to the existing system. Image review went down May 28 to June 4 (Letter June 25.)
More Venture CSS tests July 9 led to a letter to FSTech (Garrett to Thompson) July 11 saying: "several test items were not fully demonstrated and many were found to be deficient."
By one account only 2 of about 18 tests passed.
The overall demonstration of Venture CSS was inadequate and there was a lack of progress Garrett wrote.
"As previously experienced at other FAT sessions major functionality was scheduled to be demonstrated in FAT B yet was omitted as it was simply not ready. I must express a lack of confidence on my part that (Venture CSS) VCSS can be fully functional... to be approved by TxDOT and implemented on the currently scheduled go-live date."
July 10 the TOD (toll operations division) notified top TxDOT officials that Venture CSS was a flop, provided a plan for extending present VTX systems' life and recommended cancellation of the FSTech contract.
July 17 Phil Wilson executive director TxDOT wrote (copy nearby) Manfred Reitsch, president FSTech in Irvine CA and Thompson in Austin saying in two sentences that the contract was "terminated" as of 5pm local time.
Going forward, they will re-procure but no date set
TxDOT tells us they plan to continue for the time being using their existing systems.
FSTech's closely associated contract with TxDOT called Toll Operations & Customer Service Center Systems and Operations (TOCSCSO) that covers cash toll collections and management, a call center, pay-by-mail invoicing and imaging, TxTag account management, transaction processing, back office system operations and maintenance, collections, out-of-state payment processing, violations processing and toll plaza maintenance.
This contract was signed in October 2011, and continues, TxDOT say.
"We do plan to re-procure" the statewide back office system contract canceled July 17, says Mark Cross, media relations at TxDOT. But, he says they haven't yet got a date or thought through details of how the RFP will be revised.
Gary Thompson is a toll industry veteran with a business/accounting degree. A good natured bearded man about 60, he is from southern California. He joined FSTech in April 2011 after two years with ACS in the west.
2005 to 2008 he was roadway operations manager at the South Bay Expressway in the San Diego area.
2001 to 2005 he was cash operations manager at the Orange County TCAs.
1998 to 2001 he'd been manager of the Foothills and Eastern tollroad hiring and managing people as it started up.
He came to tolling from customer relations at Xerox Corp in the 1990s.
Definitions of tests:
A testing specialist provides these definitions:
FAT - Factory Acceptance Testing. The system is tested to insure all functionality is present. Open account, close account, reconcile, issue transponders, place transponder in Lost/Stolen statues and so on. Can be in a lab type environment. Does not have to interface with ultimate entities to send transactions anywhere.
POT - Prototype On-site Testing. Full functionality and all interfaces in place. Files, transactions, all other data, reconciliation and every designed functionality works as specified. This should be done on the proposed production system in its final operating place and final operational configuration. The vendor must pass this phase to move into full deployment in preparation for complete operational testing and system acceptance.
on the award of the contract in December 2011:
see correction Aug 22 on circumstances of Thompson's deparature:
TOLLROADSnews 2012-08-10 CORRECTION Aug 22