IT Horror Stories. Be Afraid. Be Very Afraid.

October 23, 2006, 04:29 PM —  BMC Software — 

The visions of axe murderers, zombies, and giant spiders that Halloween conjures are nothing compared to the very real threats to your enterprise. Do you have any idea of the potential nightmares that lay dormant within your own organization? Here are two true tales of IT terror, guaranteed to keep you up at night. Read them if you dare.

Nightmare #1 -- When change becomes your enemy

On a cold, rainy night, a rogue IT administrator in his dimly lit cubicle made a simple decision that halted business for several weeks. This administrator, we'll call him Frank N. Stein, was the last person to approve a change and he made one horrific mistake. He added a single line to the approval that said, "And change the database from 3.2 to 3.3."

That simple statement to upgrade the database to a new version was all it took to unleash pure havoc. How did this happen? Was some malicious evil at work? No, he just innocently thought, "why not get two things done at once?" But the upgrade of the database itself hadn't been approved or tested. People started moving the new version of the database into production and since this hadn't been planned, the appropriate backups were not in place.

In this terrifying scenario, transactions appeared to be running okay and people were able to enter orders, but the database wasn't being updated correctly. The front office sold the goods, the Web staff took the orders, but when the back office tried to process the orders they saw only gibberish -- the names of the customers weren't showing up and order amounts were incorrect. To make matters worse, this occurred just before Christmas!

Fortunately, this company was able to survive the fiasco, but poor Frank was never heard from again.

How can such a hair-raising situation be avoided? Well, you could implement an incredibly draconian measure that disallows any changes unless the CIO and 16 members of the board sign off on the agreement. Of course, that would also bring your business to a screeching halt.

A better option would be to implement a closed-loop change management system that goes through every step of the approval process, checks it and verifies it. This automated system would have prevented unapproved additions to a change process from happening. It would have used a Definitive Software Library (DSL) to control what software is allowed to be implemented in production. The new database version then would have only been installed or upgraded if it had been through testing and approval and added to the "golden images" in the DSL.

Nightmare #2 -- Night of the living dread

Here is another scary situation that will surely make IT professionals shake in their boots. Imagine that an IT organization for a global company had an excellent system for conducting online business transactions. It ran beautifully. In fact, it was so effective that the company's business development team decided to sell the online service for running and processing transactions to their customers, and promoted it as reliable and secure. All a customer had to do was hook into the company's systems. Sounds like a dream, right?

But here comes the really scary part. The business development group didn't tell IT what they were doing. IT was flooded with a workload they couldn't support. Orders took so long to get through that customers just abandoned the company's website. Even though the IT staff stayed up all night trying to fix the problem, they couldn't meet the performance requirements necessary to sustain this new, unplanned business.

The moral: Just as IT needs to let the business know what it's doing, the business should talk to IT. When proper communication channels are put in place, the IT organization can effectively plan for needed capacity to handle any type of workload. Incorporating best practices from the IT Infrastructure Library® (ITIL®) and adopting a Business Service Management approach will help establish those communication channels and will allow you to model different service scenarios and offer a complete view of the people, processes, and technology that support the services IT provides for the business.

Trick or Treat

The approach you take in dealing with potential IT nightmares can mean the difference between having a well-run organization that meets business objectives or one that's in danger of being given the corporate axe because of dissatisfied customers and personnel. The choice is yours.

The ideas expressed in this article are solely those of the author and do not necessarily reflect the opinions of ITworld.com.

» posted by abennett

BMC Software

I like it!
Post a comment
The content of this field is kept private and will not be shown publicly.
  • Allowed HTML tags: <a> <em> <strong> <cite> <code> <ul> <ol> <li> <dl> <dt> <dd>
  • Lines and paragraphs break automatically.
Resources
White Paper

Symantec Backup Exec 12 and Backup Exec System Recovery 8 deliver industry leading Windows data protection and system recovery. Download this whitepaper to find out the top reasons to upgrade and how to get continuous data protection and complete system recovery.

Webcast

Data and system loss — from a hard drive failure, malicious attack, natural disaster, or simple human error — can happen anytime. Don’t leave your business vulnerable. Make sure you have a secure recovery strategy in place. Symantec's latest backup and system recovery technology can efficiently restore critical applications, individual emails and documents and even restore your entire system in minutes in the event of a loss.

White Paper

Businesses face a growing challenge to ensure that the IT environment is properly protected. Backup Exec 12 integrates with other applications in the Symantec family of products, to complement your current data protection strategy, keep your data securely backed up and make it recoverable when you need it most.

Free stuff

VMware ESX Server in the Enterprise
By Edward L. Haletky
Published Dec 29, 2007 by Prentice Hall.
Enter now! | Official rules | Sample chapter

Green IT
By Toby Velte, Anthony Velte, Robert C. Elsenpeter
To be published Oct. 10, 2008 by McGraw Hill Professional
Enter now! | Official rules | About the book

Featured Sponsor

AISO founders envisioned a Web hosting company that was environmentally friendly. While the company employed energy-efficient innovations like solar panels, its infrastructure produced unacceptable power and cooling requirements. Find out how AISO leveraged AMD technology to overcome their challenge in this case study white paper.

In this whitepaper, Scalar explores the opportunity to change the landscape with respect to mission critical databases built around Oracle. Leveraging technologies such as Linux, high-end commodity processing power and Oracle RAC technology to architect, design, build and maintain database infrastructure that delivers maximum availability, reliability and performance at a fraction of traditional cost.

On a typical day, weather.com, the Web site for The Weather Channel in Atlanta, serves up between 15 million and 20 million page views. But in September 2004, when back-to-back hurricanes ransacked Florida, the peak traffic on one day more than tripled: over 70 million page views by more than 7 million unique visitors. Read the full success story now.

More Resources