Stupid user tricks 4: IT horror never ends

Nine more real-world disasters courtesy of your network's weakest link

Nothing can screw up a well-managed network faster than the people for whom you built it. Whether it's user error, optimistic expectations, or simply that bastard Murphy, IT's job is rarely predictable.

Lucky for you, there are lessons to be learned from others' misfortunes. So rather than wait to make your own forehead-shaped dent in the office wall, familiarize yourself with the screwups detailed below. It will make you that much more prepared to safeguard your IT environment from the ever-evolving boneheaded tendencies of those you serve.

Stupid user trick No. 1: Home is where the malware isIt happens at least once a year, and this year it happened twice, writes one IT admin: "And though we make the point with memos and lectures, there always seems to be someone who gives their work PC to the kids at night."

[ Users are by no means alone when it comes to hard-headedness in the IT world. See "Stupid user tricks 3: IT admin follies" and "True IT confessions" for real-world tales of folks who should know better fouling up. ]

The situation is familiar: To save on expenses, folks buy fewer home PCs, but their kids want to use them more than ever. Enter the corporate laptop into the home Web surfing environment -- a recipe for disaster for IT.

And it's not just kids playing games and doing homework. It's spouses using social networking -- and that uncle nobody talks about surfing porn on your corporate machines.

"Our security tends to be better than the average home box, but that won't protect you forever if you actually run out and look for attack sites," our admin warns. Sooner or later, one of your user's laptops will get compromised, leaving your network exposed to infection the next time he or she logs on at the office.

"We've gotten better at catching these compromised machines early, so instead of it being the big problem it used to be, last year it mainly just confirmed our investment in end-client security," the admin says.

The worst offender? A procurement manager who was found to have a keyboard logger installed on his company-issued laptop. "And this was a guy who spent several $100K a year online for the company," the admin informs us.

Solution: End-point security goes a long way toward preventing infected machines from gaining access to the corporate net, but they'll never be 100 percent effective. Web browsers are the gateway to hell when it comes to attack entry points. Let your users surf helter skelter and your attack potential goes way up. The only preventative measure: a strong fair-use policy and a management staff that'll enforce it.

Moral: Users will continue to break your official-use policy as long as money is tight and they believe the consequences are minor. Include disciplinary action in your policy, and make sure users know you're tracking Web site visits and system access. Otherwise, you are simply setting yourself up for disaster. Another solution: Sponsor employee discounts on netbooks. That way, your users will be less tempted to transform company property into their home PCs.

Stupid user trick No. 2: Message to self: E-mail isn't for everythingSometimes all it takes is a well-meaning IT management decision to set stupid users in motion, writes P. Lindo, an IT admin at a New York-based organization with more than 1,000 e-mail inboxes, which the firm first maxed out at 100MB per mailbox, then at 500MB.

"In 2007, we hired a new IT manager who got placed in charge of e-mail management," says Lindo. "He saw the load of user requests for larger mailbox space and decided this was where he was going to make a big difference."

And so he set about purchasing enough hardware to increase individual mailbox sizes to 1GB -- barely.

"He also used all the user requests to get backing to upgrade everyone to Office 2007 -- the one with the new Outlook mailbox search," Lindo says.

Throw in a new policy for teaching users proper inbox maintenance, and watch inbox utilization hover at a manageable 75 percent -- until you put policy into practice.

"Turns out users don't read documents titled 'Proper Inbox Space Management,'" Lindo says. What they see instead is the fine print that tells them they now have 1GB of mailbox space. And then they start using Outlook's handy new search feature to turn their e-mail clients into personal information managers.

"Nobody deletes attachments anymore. Instead they leave them in their inboxes so that they can run quick searches against them, where all they need to remember is a rough description of the attachment and the name of the person who might have sent it to them," Lindo explains.

Worse, they send attachments to themselves just so the doc will be in the inbox somewhere.

"Our mail servers got maxed out inside of three months."

The small saving grace?

"We actually saw a 35 percent decrease in the amount these users used their network home directories," Lindo reports. "Outlook became the main network gateway for personal storage. So we were able to repurpose some storage from the file server machines on the e-mail infrastructure, but we still had to make several large and unscheduled server purchases to keep up with new demand."

Solution: A big inbox may sound like a good idea, but proper capacity planning is an even better one. Moreover, planning for 75 percent utilization is a recipe for trouble. Instead, target 50 percent or less, or run a pilot project before committing. A low-cost SAN can help here as well; adding capacity to one of those is significantly easier than installing new servers.

Moral: If it seems like everyone's working harder these days it's because they are. Users will utilize any tool you put in front of them to get the job done. And if they're more familiar with their e-mail client than other network resources, they'll use it as a substitute -- as long as you let them. Expand your definition of "desktop management" to include reaching out to users to train them on the tools your company is spending money on.

Stupid user trick No. 3: Outsourcing Web development to the corner officeHere's a thought: Don't let the CEO design your company's customer-facing Web site just so he can save a few bucks, advises an IT consultant.

"We tried to sell a medium-sized company client on both a network install and a Web site design project," the consultant says. "We got the install contract, but the CEO figured he could design his site himself.

"When his general manager -- who was also his wife -- called us back in, she pulled the site up and it was hard not to wince. He'd used an open source editor with what looked like every freely downloadable template, fonts, and flashy widgets he could find. It looked like a teenage MySpace page."

[ More manager mishaps when meddling in IT can be found in "More stupider user tricks: IT horror stories redux" ]

Sure, the company's product information was now available on the Web, but the lack of customer-facing tools and analysis features did not bode well for the company's Web future.

"Even the Webmaster e-mail link didn't work," the consultant says. "Needless to say, the site was not attractive to customers, so Web revenue was low, and all those new and expanding Web marketing possibilities were crippled. The same CEO who built the site started spouting about how the rumors of e-commerce revenue were false."

Solution: Today, Web site design is cheap. From local outfits to eBay or Craigslist, the cost of a decently designed Web page has dropped from thousands of dollars per page to hundreds -- or less. Stop being penny-wise and pound-foolish.

Moral: Company Web sites can't be an afterthought investment, especially for small businesses. Not just an important face to your customer, your Web site is possibly the best way to analyze exactly who your customers are and how to sell to them. Treat it professionally, and you can leverage it for additional opportunities, including market research, customer analysis, and more.

Stupid user trick No. 4: Keep your enemies close, but your Linux talent closerGoing open source can save big bucks -- unless you leave your entire open source infrastructure in the hands of a single college intern, warns an admin at a small IT services firm.

"I finally find a small-business client who made the jump to Linux -- well, Linux and HP-UX due to a silo app they had to run for two big clients," the admin says. "Our new client had used his college intern to setup the basic network, but the kid had left for summer vacation a day earlier and suddenly the network was down. We were the first outfit in the phone book that didn't shy away from the phrase 'Debian on the desktop.'"

When the admin and his cohorts arrived, all the client's server lights were green, but nobody was connecting to anything and no one could log in to the system.

"We had to restore the servers from the ground up, which took about an hour. Everything was humming after that, so we took the time to sit down with the CEO and discuss plans for the network," the admin says. Stoked to locate someone unafraid to talk about open source software, the admin and his team got a little carried away shooting the bull with the CEO and stayed for more than an hour.

"As we were on the way out, the servers dumped again," the admin says. "Same story as before. Not wanting to lose our new penguin client, we rolled up our sleeves, restored the servers, and started digging for root cause."

What they found was a cron job set up off root.

"The cron 'cd'ed to a backup directory that tried to remove the files from a lengthy list of source directories, including several that didn't exist," the admin says. "Seems the kid had been changing these on the fly for some reason -- and he apparently liked doing sys admin as root. Academics."

Solution: Protect root access. Test your cron jobs. And maintain those server backup images.

Moral: Linux has definite benefits, but there's no denying that managing it requires a certain skill set. It's not something to trust entirely to an intern.

Stupid user trick No. 5: FacebookFace it, even the most stringent social networking policies can't diffuse the ticking time bomb that is Facebook. Throw in a little Jäger, some IT naivete, and you're set for devastating corporate embarrassment.

"About a year ago, I get a call from a junior VP who's yelling at me that he's desperate and needs me to do a 'recall on Facebook,'" says one admin who wishes to remain anonymous. "I try and get a word in edgewise, but he's ranting about what crap Web technology is and why computer people can't just leave well enough alone and how everything was fine when we just used the telephone. Then he ends with, 'Is it done yet?'"

"'Is what done?'"

"'The Facebook recall, for @#$%'s sake.'"

Which gave our admin the obvious pleasure of asking, "What the @#$% is a 'Facebook recall'?"

As it turns out, the junior VP had updated his Facebook page from his phone while having a few drinks with some senior VPs and potential new clients.

"He stated that he'd dated one of the clients' wives and made some nasty comment about what she looked like naked," the admin says. "All his college buddies were on Facebook in their college group, and he knew her when they were both at school. Turns out that's where she met her husband, too, and he was on the college Facebook group as well, which the genius junior VP figured out when he got back to the table and started a conversation about Facebook with the potential clients."

As for the "Facebook recall," it appears that the junior VP thought updating his Facebook page was like sending an e-mail in Outlook.

"I told him how to lock down his page, but apparently that was a little late," the admin says. "We didn't get that account."

Solution: There really isn't one, other than trying to make sure your users have some idea of where the power of IT ends and the big, bad world begins.

Moral: The beauty of social networking is that it connects you with millions of other people. The danger of social networking is that it connects you with millions of other people.

Stupid user trick No. 6: Offshoring while under the influence of MBAWhoever said offshoring was idiot-proof? After all, it often involves upper management -- potentially the worst IT offenders of all.

"We got a new CIO just before the bubble burst back in 2000," says D. Aubrey, who at that time worked at a Web services firm with a solid market position that it now had to defend against upstarts. "She was one of those MIS MBAs -- emphasis on MBA. All you press types started writing stories about the benefits of outsourcing around then, so she jumped on the trend, canned our Web dev team, and outsourced the whole shebang to an outfit in Mumbai that worked for $25 an hour."

The plan looked good on paper -- until you looked at the paper.

"We got a hold of the plan spreadsheet she presented to the CEO, and all she'd done was compare the cost of software tools and staff from in-house to out-of-house, so obviously the savings looked huge," Aubrey says.

"Then came the phone bill, which I think had quadrupled for that project," he adds. "And the security audit bill, since the data our Web dev guys were working with was quite a king-size waffle of personal customer data. And the hardware/services bill for moving our data out of the outsource outfit's internal datacenter -- which as far as we could tell was four servers in a closet somewhere -- and into a professional data hosting facility in Europe."

1 2 Page
Insider: How the basic tech behind the Internet works
Join the discussion
Be the first to comment on this article. Our Commenting Policies