IT Horror Stories. Be Afraid. Be Very Afraid.

by Peter Armstrong

October 23, 2006 —

The visions of axe murderers, zombies, and giant spiders that Halloween conjures are nothing compared to the very real threats to your enterprise. Do you have any idea of the potential nightmares that lay dormant within your own organization? Here are two true tales of IT terror, guaranteed to keep you up at night. Read them if you dare.

Nightmare #1 -- When change becomes your enemy

On a cold, rainy night, a rogue IT administrator in his dimly lit cubicle made a simple decision that halted business for several weeks. This administrator, we'll call him Frank N. Stein, was the last person to approve a change and he made one horrific mistake. He added a single line to the approval that said, "And change the database from 3.2 to 3.3."

That simple statement to upgrade the database to a new version was all it took to unleash pure havoc. How did this happen? Was some malicious evil at work? No, he just innocently thought, "why not get two things done at once?" But the upgrade of the database itself hadn't been approved or tested. People started moving the new version of the database into production and since this hadn't been planned, the appropriate backups were not in place.

In this terrifying scenario, transactions appeared to be running okay and people were able to enter orders, but the database wasn't being updated correctly. The front office sold the goods, the Web staff took the orders, but when the back office tried to process the orders they saw only gibberish -- the names of the customers weren't showing up and order amounts were incorrect. To make matters worse, this occurred just before Christmas!

Fortunately, this company was able to survive the fiasco, but poor Frank was never heard from again.

How can such a hair-raising situation be avoided? Well, you could implement an incredibly draconian measure that disallows any changes unless the CIO and 16 members of the board sign off on the agreement. Of course, that would also bring your business to a screeching halt.

A better option would be to implement a closed-loop change management system that goes through every step of the approval process, checks it and verifies it. This automated system would have prevented unapproved additions to a change process from happening. It would have used a Definitive Software Library (DSL) to control what software is allowed to be implemented in production. The new database version then would have only been installed or upgraded if it had been through testing and approval and added to the "golden images" in the DSL.

Nightmare #2 -- Night of the living dread

Here is another scary situation that will surely make IT professionals shake in their boots. Imagine that an IT organization for a global company had an excellent system for conducting online business transactions. It ran beautifully. In fact, it was so effective that the company's business development team decided to sell the online service for running and processing transactions to their customers, and promoted it as reliable and secure. All a customer had to do was hook into the company's systems. Sounds like a dream, right?

But here comes the really scary part. The business development group didn't tell IT what they were doing. IT was flooded with a workload they couldn't support. Orders took so long to get through that customers just abandoned the company's website. Even though the IT staff stayed up all night trying to fix the problem, they couldn't meet the performance requirements necessary to sustain this new, unplanned business.

The moral: Just as IT needs to let the business know what it's doing, the business should talk to IT. When proper communication channels are put in place, the IT organization can effectively plan for needed capacity to handle any type of workload. Incorporating best practices from the IT Infrastructure Library® (ITIL®) and adopting a Business Service Management approach will help establish those communication channels and will allow you to model different service scenarios and offer a complete view of the people, processes, and technology that support the services IT provides for the business.

Trick or Treat

The approach you take in dealing with potential IT nightmares can mean the difference between having a well-run organization that meets business objectives or one that's in danger of being given the corporate axe because of dissatisfied customers and personnel. The choice is yours.

The ideas expressed in this article are solely those of the author and do not necessarily reflect the opinions of ITworld.com.

BMC Software