Computer projects have failed for as long as there have been computers. But now that most companies are only as stable as their bits and bytes, the consequences of information technology screwups aren't easily disguised -- they show up in earnings reports.
When IT goes bad, high-growth rocket ships like Oxford Health Plans Inc. and Ben & Jerry's Homemade Inc. report their first-ever financial losses. Others crater and run for bankruptcy protection, as did drug distribution giant FoxMeyer Corp.
In a Computerworld study of multimillion-dollar IT disasters, the following two not-so-surprising themes emerged:
• User companies like FoxMeyer often file tough-to-win lawsuits against the vendors or consultants involved. Nonetheless, collectively, users rarely seem to learn much from the episodes or apply the lessons to future projects.
• All of the botched projects in Computerworld's Top 10 disasters list were big and richly complex; many were the toughest IT projects the users had ever tried. Five were hideously difficult enterprise resources planning (ERP) system implementations.
Root Causes Remain the Same
The root causes of IT failures haven't changed a bit over the years.
Miscommunication, hazy goals, "scope creep," inept leadership, pitiful project management -- you've heard, if not heeded, it all before.
"We may be neck-deep in the New Economy and Internet time, but you still have the same factors and the same failings," said Bruce Webster, a director at PricewaterhouseCoopers in Washington.
Webster recently studied 120 IT lawsuits filed since 1976, and he said he's convinced that most flops could be avoided if people simply knew the time-honored best practices of systems development.
Kevin Hickey was hired to clean up the IT mess at Oxford Health Plans
"I don't know how many IT managers, team leaders, directors and CIOs have actually sat down and read The Mythical Man-Month, The Psychology of Computer Programming and Death March," he said, referring to three books that amount to the software development canon. "The causes of disasters are all well documented. They're fundamental."
Still, warning lights are easy to overlook when the whole room is spinning.
"There's a natural tendency to get overly committed to something, especially when there are no clear signals telling you you are off course," said Mark Keil, an associate professor at Georgia State University in Atlanta.
The infamously buggy baggage-handling system at the Denver International Airport is one case that offered unambiguous proof of technology glitches: shredded luggage.
But tests of most questionable IT projects don't yield such graphic evidence.
In large systems integration or ERP deals, "there's no torn suitcase sitting at your feet to wake you up," said Keil, who has studied IT disasters for nine years. "So it's a lot easier to delude yourself into thinking things aren't that bad."
Project Euthanasia
Top 10 Failures
See Top 10 Corporate Information Technology Failures in a PDF chart. (Requires Adobe Acrobat Reader)
Euthanasia for the project might be the best course, but people often have too much heart and money invested to end it.
One technique for preventing a disaster is to add some humility to the endeavor. Invite a third party to review your work - a reliable consultant, an academic or a buddy CIO.
Dishonorable Mentions
Some information technology disasters of the past decade aren't included in the Top 10 chart (at right) because their financial costs weren't clear or weren't quite as high as those that made the list. Others were quasigovernmental. But they were significant bungles nonetheless. Here's a sampling:
• High hopes for a high-tech airport in Denver were dashed in 1994 when a baggage-sorting system misplaced and damaged countless suitcases. The object-oriented system, which was built by BAE Automated Systems Inc. in Carrollton, Texas, ran on IBM's OS/2.
To fix it, primary sponsors Chicago-based United Air Lines Inc. and the city of Denver ended up paying at least $86 million more than the original $193 million price tag. The airport opened almost three years late.
• Ben & Jerry's Homemade Inc., a small Vermont ice cream maker that hit the big time in the early 1990s, took a $4.1 million write-off in 1995 when it canceled a project to build a fully automated packing plant. As a result, the company posted its first quarterly loss ever.
• Early this year, Thomas & Betts Corp., a $2.5 billion electrical parts maker in Memphis, blamed problems with a new Internet-based order-management system for a 50% drop in profits in the fourth quarter of last year.
The homegrown mainframe system also cost the company another $42 million in order and shipping disruptions. In the following quarter, the company spent $11 million on customer service, extra freight costs and other measures to make up for ongoing system crashes and backlogs.
In a lawsuit still pending, shareholders say Thomas & Betts misled them about the success of the new system.
• Just in time for the Memorial Day holiday crush in 1998, Avis Inc.'s Wizcom reservation system crashed. The outage blocked the rental car company, as well as many travel agencies and hotels also linked to the system, from taking bookings for 30 hours.
• In the summer of 1994, an error in a routine file update crashed the automated teller machine (ATM) system of Chemical Banking Corp. in New York. ATMs at Chemical, which was then one of the five largest banks in the U.S., were down for five hours.
• In 1993, Fidelity Brokerage Services Inc. in Boston started a multimillion-dollar project to build a Windows-based trading application for customers with home PCs. "Jamaica" still wasn't ready by mid-1996, reportedly because of political clashes between quirky programmers and staid investment bankers.
The application wasn't a failure; it worked when it was finally rolled out. But the numerous development delays put Fidelity far behind rivals such as Charles Schwab & Co. in San Francisco.
An outsider can "walk into the project setting for 20 minutes, talk to a few people and come to the conclusion that things have run amok. But people inside may not even be aware," Keil said.
Greyhound Lines Inc. in Dallas, for example, seemingly didn't know anything was wrong with its new reservations and logistics system -- until it went live and 12% of its customers went away mmad in one month.
Though specific individuals might learn from their own mistakes, those lessons aren't transferred to any collective IT consciousness.
"The people with that [failure] experience aren't always the people in authority the next time that situation arises," observed Kevin Hickey, a former head of IT at Trumbull, Conn.-based Oxford Health Plans Inc. "The fact is, hubris will always be with us."
And then there's what Webster calls "the thermocline of truth." Swimmers know that lake water separates into warm and cold horizontal bands. The area between is a thermocline.
In IT groups, everyone below Webster's "thermocline of truth" knows the project is sinking, while everyone above it thinks things are fine. Senior executives can be oblivious. They aren't involved enough, they don't want to have to face a failure, or underlings are afraid to tell them, he explained.
"You can see that persist almost until the point where the project is supposed to be delivered," he said. "Then, suddenly, it's, 'What do you mean this will take another six months?' "
That was part of the sorry plight of Fort Worth, Texas-based AMR Corp.'s Confirm reservations system. Confirm managers are even said to have orchestrated a cover-up.
Overall, IT culture is such that problems, especially expensive ones (which hold the most valuable lessons), are hidden. Programmers write around buggy code rather than tear it apart. Managers revise project specifications to reflect what they did instead of what they should have done. Senior IT leaders neglect to tell their bosses the bad news.
Most companies are too embarrassed to analyze their failures, said Effy Oz, an associate professor of management and IT at Pennsylvania State University in Great Valley.
"People will say, 'There's no time, and we're not paid to have these discussions,' " Oz said. "The CEO has to be a very confident person to say, 'These things happen. Let's learn from it.' "
The average loss in an abandoned project is $4.2 million, according to Oz. The blowups in Computerworld's top 10 list cost much more than that. And, if history is any indication, they will happen again.
Travel System Is Confirm-ed Disaster
On the heels of its hugely successful Sabre airline reservation system, Fort Worth, Texas-based AMR Corp., the parent company of American Airlines Inc., in the late 1980s formed a joint venture with Marriott International Inc., Hilton Hotels Corp. and Budget Rent A Car Corp. to build a similar system for the travel industry. But Confirm, as the project was called, wasn't to be. In fact, the effort is viewed as one of the worst IT failures ever for its mismanagement, questionable ethics and unworkable software.
AMR's information systems unit in Dallas was the lead developer on Confirm, which was originally due in June 1992 for no more than $55.7 million. Yet Confirm started to miss deadlines and cost estimates several months after development began.
Specifications were unclear, unskilled programmers were assigned to the project and mainframe-based transaction-processing software couldn't be integrated with a related mainframe decision-support system. Moreover, one year into design and development, Confirm had already fallen one year behind schedule.
Bethesda, Md.-based Marriott and Lisle, Ill.-based Budget started asking questions in 1990 but were assured that Confirm would work and that programmers would make up time and meet the deadline.
In April 1992, just three months before it was slated to go live, Confirm failed tests at Los Angeles-based Hilton. AMR also told its partners in a letter that it needed another 15 to 18 months.
"The individuals whom we gave responsibility for managing Confirm have proven to be inept [and] concealed a number of important technical and performance problems," the AMR letter said.
The legendary Max Hopper,, who had been instrumental in Sabre's development, was vice chairman of AMR's IT unit at the time, though not directly involved in the daily work of Confirm. However, he acknowledged in a note to his employees that some Confirm managers "did not disclose the true status of the project in a timely manner. This has created more difficult problems -- of both ethics and finance -- than would have existed if those people had come forward with accurate information" [News, May 22, 1995]. After consuming almost four years and $125 million, Confirm was effectively dead.
In September 1992, AMR sued Budget, Hilton and Marriott; Marriott then sued AMR. The suits were settled out of court for undisclosed terms. Hopper recently declined to discuss Confirm, citing the secret settlements.
AMR was mainly to blame, but all sides acted unprofessionally, said Effy Oz, an associate professor of management and IT at Pennsylvania State University.
Executives at AMR "simply lied to their client-partners," said Oz, who has studied Confirm. "The partners were irresponsible in not asking more questions more often and in accepting at face value all that [AMR] told them."
Budget, Hilton and Marriott today use their own -- separate -- reservation systems.
A Really Bad Bet for Drug Distributor
When it launched a $35 million enterprise resource planning (ERP) project way in 1993, FoxMeyer Corp. was a $5 billion drug distributor in Carrollton, Texas.
Now it's bankrupt.
It wasn't the fumbled IT project alone that destroyed FoxMeyer, but that was a critical contributor, according to lawsuits FoxMeyer filed against SAP AG, SAP America Inc. and Chicago-based Andersen Consulting in 1998.
SAP lied repeatedly about R/3's capabilities, and Andersen's inexperienced staff used FoxMeyer as a training camp, claims the drug company, which seeks damages of $500 million from SAP and $500 million from Andersen.
FoxMeyer was one of the first big users to take a chance on ERP, which was a hot new idea at the time. Perhaps the company should have been more cautious. Legal documents show that FoxMeyer knew that SAP's R/3 software hadn't yet been used at distribution companies -- just at manufacturers.
Still, FoxMeyer was jazzed. Then-CIO Robert Brown told Computerworld in 1994, "We are betting our company on this."
Big problems started to emerge later that year. For example, the new R/3 software miscounted inventory, which in turn screwed up customer orders. Outright crashes were routine.
SAP declined to talk specifically about FoxMeyer. But an SAP spokesman said users who install R/3 are usually changing basic business processes at the same time, which "is typically where most of the pain and challenges of implementation come from."
FoxMeyer also charges that R/3 performed worse than the company's proprietary Unisys Corp. system. R/3 could process just 10,000 invoice lines per night, compared with 420,000 for the Unisys setup.
SAP misled FoxMeyer with faulty benchmarks, according to the suit. Other users have also questioned SAP's benchmarks [News, Sept. 4, 1995]. SAP doesn't misrepresent benchmarks, the company's spokesman said.
As for Andersen, its people "were neophytes," said Mark Ressler, a lawyer from New York firm Kasowitz, Benson, Torres & Friedman LLP who represents FoxMeyer.
According to FoxMeyer, many Andersen workers were recent college graduates, and others lacked R/3 experience. "There's no better way to overcome that than [by] experimenting on a live patient," Ressler said.
They bungled data conversions by using incorrect drug product codes, for example, and built faulty interfaces between the old and new systems, the lawsuit charges.
As a result, most R/3 modules were rolled out to just six of 23 warehouses by the time the company filed for bankruptcy protection in August 1996.
Andersen didn't respond tto requests for comments. The suits are slated for trials next May.
Meanwhile, FoxMeyer itself is being sued by stockholders, in part for allegedly hiding timely information about the computer problems and their effects on business.
High-Flying HMO Modernizes, Crashes
Oxford Health Plans Inc. was the Netscape of health maintenance organizations. It seemed to burst from nowhere, captivate customers and force competitors to change the way they operated.
Then Oxford decided to modernize its information technology.
It was 1995 and Oxford's old turnkey, Pick-based billing and membership tracking system would no longer do. A complete overhaul, using more modern Unix technology, is what the company wanted -- and fast.
The project included many custom applications that ran with Oracle databases and other software. But key was a claims processing system, dubbed Pulse, that Oxford's internal IT people built with Oracle tools.
Trouble hit the Trumbull, Conn.-based HMO almost as soon as the rollout started in late 1996. Customers suddenly got claims laden with errors -- when they got claims at all. The company paid bills it shouldn't have and denied claims it should have paid.
All the late and inaccurate paperwork caused New York state to fine Oxford $3 million for violating insurance laws.
Overall, the new software overestimated revenue by $392 million for 1997 and 1998 while also underestimating medical costs. That awful combination led to Oxford's $291 million loss in 1997.
Angry doctors and patients abandoned the HMO. Membership dropped 20% from 1997 to 1999, partly because of the systems problems and partly because Oxford withdrew from four of the seven states it did business in.
Ultimately, top executives left, and Oxford hired a new head of operations -- Kevin Hickey, then an operations manager at Aetna Inc. -- to help with an IT cleanup already under way.
Hickey immediately faced down the billing mess, which "wasn't just an inconvenience. This was a survival issue," he said in an interview.
First, he shelved Pulse and returned to the old Pick application. Pulse was never fully integrated with the Oracle software, he said.
Cambridge Technology Partners Inc. and Diamond Technology Partners Inc. came in for quick-hit assignments to help fix claims processing.
Oxford also shut down its advanced technology unit. Studying artificial intelligence software for possible future systems was frivolous now that "emergency" IT problems threatened to incapacitate the HMO, Hickey explained.
Oxford hired Computer Sciences Corp. to create a plan for outsourcing its entire IT operation.
But in 1998, a new CEO swept in and swept away that idea, along with most remaining legacy executives. Hickey, too, was replaced after just a year at the company.
Today, Oxford is smaller and smarter. It wrote off $5 million for hardware and software in 1998. Late last year, system fixes even took precedence over customer acquisition, according to documents filed with the Securities and Exchange Commission (SEC).
Oxford has since completed major systems fixes and put in place new quality assurance and other testing programs. But SEC documents warn that unexpected sales calculations could still turn up.
Oracle's Rotten Idea
Tri Valley Growers got caught in an Oracle Corp. software scheme that Oracle CEO Larry Ellison later admitted was a bad idea.
In November 1996, the $800 million agriculture cooperative in San Ramon, Calif., bought $6 million worth of services and software from Oracle. The core of the deal was Oracle CPG, ERP software for consumer packaged goods companies.
While the financial and manufacturing pieces of the CPG suite were from Oracle, order processing, production planning and other packages were from other vendors. Oracle consultants were hired to integrate it all. <
The new software was expected to make Tri Valley more efficient, improve customer service and save it $5 million a year.
In a press release trumpeting the Tri Valley deal in 1998, for example, Oracle said Tri Valley customers would be able to file a single purchase order, no matter how the cooperative managed its many fruit product lines internally.
But the system never got that far. Oracle couldn't get most of its applications to work with the non-Oracle software.
Tri Valley claims to have spent more than $22 million before halting the project and turning to SAP AG software.
The Oracle software "never worked, has not and cannot be integrated and could not even be installed on Tri Valley's computers," according to a lawsuit the cooperative filed in February.
Last year, Ellison called the Oracle CPG bundle "a huge mistake" [News, April 21, 1999]. Oracle no longer sells it.
In July, Tri Valley filed for bankruptcy protection. So far, it hasn't blamed its poor financial shape on the failed Oracle project - but it might, and it might ask for more money, to boot, said Peter Sipkins, Tri Valley's lawyer, with Dorsey & Whitney LLP in Minneapolis.
Oracle has denied all allegations. Tri Valley IT officials didn't return calls seeking comment. But a former IT manager there called the situation "very tragic."
Print
Close Window