It's a CIO's worst nightmare: You get a call from the Business Software Alliance (BSA), saying that some of the Microsoft software your company uses might be pirated.
You investigate and find that not only is your software illegal, it was sold to you by a company secretly owned and operated by none other than your own IT systems administrator, a trusted employee for seven years. When you start digging into the admin's activities, you find a for-pay porn Web site he's been running on one of your corporate servers. Then you find that he's downloaded 400 customer credit card numbers from your e-commerce server.
And here's the worst part: He's the only one with the administrative passwords.
Think it can't happen? It did, according to a security consultant who was called in to help the victim, a $250 million retailer in Pennsylvania. You never heard about it because the company kept it quiet.
Despite the occasional headlines about IT folks gone rogue (remember Terry Childs, the network administrator who held the city of San Francisco's network hostage?), most companies sweep such situations under the rug as quickly and as quietly as possible.
An annual survey by CSO magazine, the U.S. Secret Service and CERT (a program of the Software Engineering Institute at Carnegie Mellon University) routinely finds that three quarters of companies that are victimized by insiders handle the matter internally, says Dawn Cappelli, technical manager of CERT's threat and incident management team. "So we know that [what's made public] is only the tip of the iceberg," she says.
By keeping things quiet, however, victimized companies deny others the opportunity to learn from their experiences. CERT has tried to fill that void. It has studied insider threats since 2001, collecting information on more than 400 cases. In its most recent report, 2009's "Common Sense Guide to Prevention and Detection of Insider Threats" (download PDF), which analyzes more than 250 cases, CERT identifies some of the most common mistakes companies make: inadequate vetting during the hiring process, inadequate oversight and monitoring of access privileges and overlooking of red flags in behavior.
But threats from privilege-laden IT employees are especially hard to detect. For one thing, staffers' nefarious activities can look the same as their regular duties. IT employees routinely "edit and write scripts, edit code and write programs, so it doesn't look like anomalous activity," Cappelli says. And they know where your security is weakest and how to cover their tracks. You can't rely on technology, or any single precaution to protect yourself from rogue IT people. You have to look at the big picture.
"It requires not only looking at what they are doing online but also what's happening in the workplace," says Cappelli. "People really need to understand the patterns here, the story behind the numbers."
Computerworld went looking for some of those stories behind the numbers, incidents that have not been widely reported. Though the victimized companies wouldn't talk, the security consultants who helped clean up the messes would. Although each story has unique circumstances, together they show some of the typical patterns that CERT emphasizes. Employer, beware.
Pirating software -- and worse
The Pennsylvania retailer's tale of woe began in early 2008, when the BSA notified it that Microsoft had uncovered licensing discrepancies, according to John Linkous. Today, Linkous is chief security and compliance officer at eIQ Networks, a security consultancy. His experience with the incident involving the retailer is from his previous job, when he was vice president of operations at Sabera, a now-defunct security consultancy.
Microsoft had traced the sale of the suspect software to a client company's sysadmin. For purposes of this story, we'll call that sysadmin "Ed." When Linkous and other members of the Sabera team were secretly called in to investigate, they found that Ed had sold more than a half-million dollars in pirated Microsoft, Adobe and SAP software to his employer.
The investigators also noticed that network bandwidth use was abnormally high. "We thought there was some kind of network-based attack going on," says Linkous. They traced the activity to a server with more than 50,000 pornographic still images and more than 2,500 videos, according to Linkous.
In addition, a forensic search of Ed's workstation uncovered a spreadsheet containing hundreds of valid credit card numbers from the company's e-commerce site. While there was no indication that the numbers had been used, the fact that this information was contained in a spreadsheet implied that Ed was contemplating either using the card data himself or selling it to a third party, according to Linkous.
The CFO, who had originally received the call from the BSA, and others on the senior management team feared what Ed might do when confronted. He was the only one who had certain administrative passwords -- including passwords for the core network router/firewall, network switches, the corporate VPN, the HR system, the e-mail server administration, Windows Active Directory administration, and Windows desktop administration.
That meant that Ed could have held hostage nearly all the company's major business processes, including the corporate Web site, e-mail, financial reporting system and payroll. "This guy had keys to the kingdom," says Linkous.
So the company and Linkous' firm launched an operation right out of Mission: Impossible. They invented a ruse that required Ed to fly overnight to California. The long flight gave Linkous' team a window of about five and a half hours during which Ed couldn't possibly access the system. Working as fast as they could, the team mapped out the network and reset all the passwords. When Ed landed in California, "the COO was there to meet him. He was fired on the spot."
Cost to the company
Linkous estimates that the incident cost the company a total of $250,000 to $300,000, which includes Sabera's fee, the cost of flying Ed to the West Coast on short notice, the cost of litigation against Ed, the costs associated with hiring a temporary network administrator and a new CIO, and the cost of making all of its software licenses legitimate.
What could have prevented this disaster? Obviously, at least one other person should have known the passwords. But more significant was the lack of separation of duties. The retailer had a small IT staff (just six employees), so Ed was entrusted with both administrative and security responsibilities. That meant he was monitoring himself.
Separating duties can be a particularly tough challenge for companies with small IT staffs, Linkous acknowledges. He suggests small companies monitor everything, including logs, network traffic and system configuration changes, and have the results evaluated by someone other than the system administrator and his or her direct reports. Most important, he says, is to let IT people know that they are being watched.
Second, the company failed to do a thorough background check when it hired Ed. In CERT's research, 30% of the insiders who committed IT sabotage had a previous arrest history. In fact, any kind of false credentials should raise a red flag. Although the company had done a criminal background check on Ed (which was clean), it did not verify the credentials on his résumé, some of which were later found to be fraudulent. (He did not, for example, have the MBA that he claimed to have.)
Third, Ed's personality could have been viewed as a red flag. "He seemed to believe that he was smarter than everyone else in the room," says Linkous, who met Ed face-to-face by posing as an ERP vendor before the sting operation. Ed's arrogance reminded Linkous of the infamous Enron executives. "He was extremely confident, cocky and very dismissive of other people."
CERT has found that rogues often have prickly personalities. "We don't have any cases where, after the fact, people said, 'I can't believe it -- he was such a nice guy,'" says Cappelli.
Outsourcing incenses employee
"Sally," a systems administrator and a database manager, had been with a Fortune 500 consumer products company for 10 years and was one of its most trusted and capable IT workers, according to Larry Ponemon, founder and chairman of the Ponemon Institute, an IT security research firm.
She was known as a pinch-hitter -- someone who was able to help solve all kinds of problems. For that reason, she had accumulated many high-level network privileges that went beyond what her job required. "There is this tendency to give these people more privileges than they need because you never know when they'll need to be helping someone else out," says Ponemon.
She sometimes worked from home, taking her laptop, which was configured with those high-level privileges. The company's culture was such that IT stars like Sally were given special treatment, says Ponemon. "The IT people made an end run around certain policies," he says. "They could decide what tools they wanted on their systems."
But when the corporation decided to outsource most of its IT operations to India, Sally didn't feel so special. Although the company had not yet formally notified the IT staff, says Ponemon, it was obvious to IT insiders that time was running out for most of the department's employees.
Sally wanted revenge. Before she was officially let go, she planted logic bombs that caused entire racks of servers to crash once she was gone.
At first, the company had no clue what was going on. They switched to their redundant servers, but Sally had planted bombs in those as well. The company had a hard time containing the damage because it didn't follow any apparent rhyme or reason. "A malicious employee [who's] angry can do a lot of damage in a way that's hard to discover immediately and hard to trace later," Ponemon notes.
Eventually, they traced the sabotage to Sally and confronted her. In return for Sally's agreement to help fix the systems, the company did not prosecute her. In addition, Sally had to agree never to talk publicly about the incident. "They didn't want her going on Oprah and talking about how she broke the backbone of a Fortune 500 company."
Cost to the company
The estimated total cost to the company: $7 million, which includes $5 million in opportunity costs (downtime, disruption to business and potential loss of customers) and $2 million in fees for forensics and security consultants, among other things.
What did the company do wrong? First, the incident is a classic example of "privilege escalation," which is what happens when privileges are granted to an individual to handle a specific task but are not revoked when the person no longer needs them, says Ponemon.
Second, an entitlement culture led to no separation of duties and very little oversight of IT. Because of that, management missed an important red flag. After the incident, the company discovered that Sally had "lost" 11 laptops over the previous three years. The help desk staff was aware of this, but no one ever reported it to management, partly because of Sally's status in the organization. Nobody knows what she did with those laptops; it could be that she was just careless -- but "that's a problem in and of itself if you're a systems administrator," Ponemon observes.
Third, given the tense atmosphere created by the outsourcing decision, the company should have been more vigilant and more proactive in monitoring potentially angry employees.
Even if you haven't announced anything to your employees, it's a mistake to think they don't know what's going on, says Ponemon. "The average rank-and-file [worker] knows within a nanosecond of when the CEO signs the [outsourcing] contract," he says. If you aren't already monitoring your IT people, now is the time to start. For best results, kick off the program with a very public pronouncement that you are now monitoring the staff.
According to CERT, many cases of sabotage are the result of a disgruntled employee acting out of revenge. And those acts can happen in the blink of an eye, as the next story illustrates.
A firing gone very wrong
When this Fortune 100 company upgraded its security, it made a nasty discovery. One of its senior system admins, who had been there at least eight years, had surreptitiously added a page to the company's e-commerce Web site. If you typed in the company URL followed by a certain string of characters, you got to a page where this admin, whom we'll call "Phil," was doing a brisk business selling pirated satellite TV equipment, primarily from China, according to Jon Heimerl, director of strategic security for Solutionary, a managed security services provider hired to address the problem.
The good news: improved security caught the perpetrator. The bad news: A flawed firing procedure gave him the opportunity to take a parting shot.
Itself a retailer in high-tech equipment, the company wanted to get rid of Phil and the Web site as quickly as possible because it feared lawsuits from satellite equipment manufacturers. But while Phil's manager and security staffers were on their way to his office, a human resources representative called Phil and told him to stay put. Heimerl isn't sure exactly what the HR person said, but it was apparently enough for Phil to guess that the jig was up.
Already logged into the corporate network, he immediately deleted the corporate encryption key ring. "As he was hitting the delete key, security and his manager showed up and said, 'Stop what you're doing right now, and step away from the terminal,'" according to Heimerl. But it was too late.
The file held all the encryption keys for the company, including the escrow key, a master key that allows the company to decrypt any file of any employee. Most employees kept their own encryption keys on their local systems. However, the key ring held the only copies of encryption keys for about 25 employees -- most of whom worked in the legal and contracts departments -- and the only copy of the corporate encryption key. That meant that anything those employees had encrypted in the three years since they had started using the encryption system was permanently indecipherable -- and thus, virtually lost to them.
Cost to the company
Heimerl hasn't calculated how much money the incident cost the company, but he estimates the loss of the key ring file amounted to about 18 person-years of lost productivity, which takes into account both the work that went into creating files that are now permanently encrypted and the time devoted to re-creating materials from drafts, old e-mails and other unencrypted documents.
Focusing only on what happened after they discovered the rogue Web site, the company made two crucial mistakes, says Heimerl. It should have shut down Phil's access immediately upon discovering his activities. But managers also left themselves vulnerable by not keeping a secure backup of critical corporate information. (Ironically, the company thought the key ring was so sensitive that no copies should be made.)
The best defense is multipronged
The overall lesson from these horror stories is that no one single thing can protect you from rogue IT people. You might have great technical security -- like the multitiered security system that ultimately detected Phil's unauthorized Web site -- and yet a simple mistake by HR can lead to disaster. There could be big red flags in terms of behavior or personality that go unnoticed -- like Sally's missing laptops.
It's a combination of technical safeguards and human observation that offers the best protection, says CERT's Cappelli.
And yet it's hard to convince companies to do both. Executives tend to think such problems can be solved by technology alone, at least partly because they hear vendors of monitoring tools and other security-minded software claiming that their tools offer protection. "We're trying to figure out how to get the message to the C-level people that this is not just an IT problem," she says.
It's a difficult message to hear. And a lesson that many companies don't learn except the hard way. Even if more companies were forthcoming with the details of their horror stories, most CEOs would still think it could never happen to them. Until it does.
Frequent Computerworld contributor Tam Harbert is a Washington, D.C.-based writer specializing in technology, business and public policy. She can be contacted through her Web site, TamHarbert.com.
This story, "Security fail: When trusted IT people go bad" was originally published by Computerworld.