The seven dirtiest jobs in IT

Working in IT isn't always pretty. After all, we can't all work on the cutting-edge technologies all the time. Some of us have to get dirty -- in some cases, literally.

Unfortunately, dirty jobs -- whether you're being chained to a help desk, hacking 30-year-old code, finding yourself wedged between warring factions in the conference room, or mucking about in human effluvia -- are necessary to make nearly every organization tick. (Well, maybe not the human effluvia part.)

The good news? Master at least one of them, and you're pretty much guaranteed a job with somebody. We don't guarantee you'll like it, though.

Here are seven of the dirtiest jobs in IT, and why your organization needs them.

Dirty IT job No. 7: Legacy systems archaeologist

WANTED: INDIVIDUALS FAMILIAR WITH 3270, VAX/VMS, COBOL, AS/400, AND OTHER LEGACY SYSTEMS NO ONE ELSE REMEMBERS. MUST BE ABLE TO TYPE ENTIRELY IN CAPITAL LETTERS FOR EXTENDED PERIODS. APPLICANTS MUST MEET MINIMUM AGE REQUIREMENT OF 55.

Believe it or not, COBOL developers are still in demand, says Jim Lanzalotto, vice president of Yoh, a technology talent and outsourcing firm.

"I'm looking at a job listing right now for a PeopleSoft business analyst," says Lazalotto. "Buried in the middle of the description, it says, 'writes COBOL as needed.' Here's another one, for a senior program analyst with a background in IBM WebSphere, EDI, Unix, and secure file transfer protocol -- 'knowledge of COBOL a plus.' Imagine your average 29-year-old hipster applying for one of these jobs. 'You want me to know what?'"

You'd think these old systems would have died off years ago, but larger companies -- especially in financial services, manufacturing, retail, and health care -- cling to them like drunken sailors to a lamppost.

"I know of at least one major office supply retailer that powers its site by connecting AS400s to Web front ends," says Andrew Gelina, CEO of Syrinx Consulting, in Waltham, Mass. "The cost of rewriting or migrating these apps is huge and the risk is high, so they look for any way possible to reuse and reconnect to modern technologies. It's like marine archeology. You'll need a spelunker to dive deep into them, figure out how they can be bolted and duct-taped into a more modern integration engine, like a SOAP/XML front end."

The good news? Experienced techs willing to do these dirty jobs may discover reliable income streams as they ease into semi-retirement.

"There's an interesting inversion principle at work here," Gelina says. "The value of people with skills built around those systems had been going steadily down for a long time. Now that companies can't find anyone to work on them, the reverse is true. If you're a consultant who specializes in one of those older technologies, you've got a pretty good niche."

Dirty IT job No. 6: Help desk zombie

Excellent entry-level opportunity for multitasking individual with low self-esteem. Ability to read from scripts a plus. Potential to move up to bug scraper, password reset technician, or tape rotation coordinator.

Here's the job that every IT professional hates. Bruce Kane, senior consultant at M3 Technology Group in Charlotte, N.C., defines a dirty job as "anything where you have to visit or talk to end-users. Help desk, desk side support, etc. Icky! Users have cooties!"

Of course, users often feel the same way about support techs, says Kris Domich, principal datacenter consultant at Dimension Data.

"When you contact tech support, a lot of people feel like they're either talking to an idiot or being treated like one," Domich says. "There's a fine line between being courteous and being patronizing, and many techs don't know where that line is."

As more organizations move to 24/7 operations, they may also need the services of the more specialized Graveyard Support Vampire, who shuns the daylight and lives by the glow of the network console.

"Why this person actually wants to forge his or her days for the joy of nocturnal employment is a dark, dark mystery that shall forever span the vast expanse of space and time," says Lawrence Imeish, principal consultant for Dimension Data's Converged Communications Group. "But it's often imperative that IT folks manage their equipment off-hours so as to avoid impact on day-to-day business activities on their networks. System reboots, patch applications, and troubleshooting also typically occur after-hours and could be a cause for system failure in and of themselves if not properly addressed during the evening hours."

Dirty IT job No. 5: On-site reboot specialist

Seeking individuals for on-site support of end-users. Must be familiar with three-fingered Ctrl-Alt-Del salute and power cord reconfiguration. Ability to withstand a variety of environments and personality types; concealed-weapons permit a plus. Individuals with anger management issues need not apply.

Closely related to the help desk zombie, but even lower on the totem pole, is the on-site reboot specialist. Unlike help desk or support vampires, the on-site rebootnik must venture out into the physical world and deal with actual people.

Joel Bomgar worked his way through college as an on-site support specialist. He recalls hot sticky summers spent driving Mississippi back roads in 100-degree heat, providing "sweatnical" support to clueless end-users.

"First there's the heat," Bomgar says. "Then you show up at the customer site, and the server room is a closet. Loud, dusty, dingy, and there's nowhere to sit down. You end up standing wedged between the server and the wall for hours at a time. It's like flying on a regional jet. Everything about it is uninviting."

It was this experience, Bomgar says, that ultimately inspired him to start Bomgar Corp. (formerly Network Streaming), a Ridgeland, Miss.-based provider of remote service solutions for SMBs. By adding a Bomgar Box appliance to a company's network, remote technicians anywhere in the world can access an end-user's PC and troubleshoot it.

Providing non-site support also puts some welcome physical distance between techie and customer.

"What makes on-site support dirty is interfacing with the user," Bomgar says. "People's workstations are often a nightmarish wreck. They issue you into a tiny room covered with dust, grit, and grime. The keyboard's broken and the mouse doesn't work, but they're used to it."

For their part, customers don't have to stop working while the tech takes over their machines (or stand near some college kid who's just been driving in 100-degree heat).

"Tech support becomes so much cleaner when you don't have to go deal with all those environmental variables," Bomgar says. "You get to interface with the technology without the grit, grime, and dirt associated with support."

Dirty IT job No. 4: Interdepartmental peace negotiator

Looking for self-starter skilled at moderating tech disputes between warring factions within the same company or between company and its client. Must possess experience in ego-stroking, manipulative massage, and hand-to-hand combat.

Cats and dogs, Democrats and Republicans, Martians and Venutians -- they're downright chummy compared to warring departments within many enterprises. Unfortunately, at some point they've got to pull together for the good of the company. That's when you call in the negotiator to smooth ruffled feathers and break up the fights.

This comes up a lot when different sides of the organization need to collaborate on, say, a company wiki, intranet, or portal, says Syrinx's Gelina. Somebody's got to play the heavy -- what Gelina calls the "Portal Majority Whip" -- to keep everyone on the same page, following the same rules.

"If someone is not riding herd on this, renegade elements will crop up and threaten the stability and usability of the portal," Gelina explains.

The problem? "The IT people want centralized control, while the typical users want to move forward at the speed of business without restrictions," Gelina says. "They don't want to have to wait for IT to decide something before they can move. Satisfying those two camps can be tough."

But this battle isn't always geeks against suits; sometimes it's geeks vs. geeks. Dimension Data's Domich likens the job to herding kittens.

"The Cat Herder has to keep multiple architect-level technologists focused on a common cause to a common problem," Domich says. "Even if it seems redundant at times, it's essential to have systems -- or, in this case, people -- in place to keep technology architects on schedule."

Domich adds, "IT project managers are gluttons for punishment." As if we didn't know.

Dirty IT job No. 3: Enterprise espionage engineer (black ops)

Seeking slippery individuals comfortable with lying, cheating, stealing, breaking, and entering for penetration testing of enterprise networks. Requirements include familiarity with hacking, malware, and forgery; must be able to plausibly impersonate a pest control specialist or a fire marshal. Please submit rap sheet along with resume.

Social engineer, con artist, penetration tester, or white hat hacker -- whatever you call it, Jim Stickley has a dirty job that actually sounds like fun. As VP of engineering and CTO of TraceSecurity in Baton Rouge, La., Stickley gets to talk his way into a client's offices, sneak into their datacenters, make off with the company's vitals, then come back later to show them where their internal security broke down.

The best part? He gets to wear disguises. Pest control specialist, AC repairman, OSHA inspector -- Stickley and his crew have a closet full of uniforms. But fireman is a particular favorite. "At one place you're the fire inspector, and girls fall all over you," Stickley says. "The next place you're wearing the pest inspector suit and you're the scum of the earth."

First, Stickley and his team take over the company's e-mail system and schedule an appointment. Then they show up in the appropriate fake attire. Whoever has been assigned to watch them usually leaves after about five minutes, Stickley says. If not, they send her out to get them coffee or offer to show her a (fake) dead mouse they found in the corner. That usually does the trick.

Once she's gone, they sneak into the security room and take all the backup tapes, load Trojans onto the servers, or plug wireless devices into the network and hack it from the parking lot.

"If we can get the backup tapes, we're done," Stickley says. "Every piece of data you'd want -- mothers' maiden names, Social Security numbers -- is on those tapes. We've also walked out with computers, boxes filled with loan documents, and applications for patents that have been drawn up but not submitted. It's amazing."

Stickley says he's penetrated more than 1,000 locations and has yet to be thwarted. The dirty part: Coming back the next day to face the people you just 0wned.

"You feel dirty, if nothing else," Stickley says. "People come up to you and they're mad. 'I can't believe I got you a cup of coffee.' But ultimately you're just trying to help them out. Nobody gets fired for screwing up. The whole point is to learn from the experience."

There's at least one person who doesn't gain much from Stickley's exercise in creative insecurity, however.

"I feel really bad for the real pest inspector," he says. "The next time he shows up, boy does he get beaten down."

Dirty IT job No. 2: Datacenter migration specialist

Position involves relocating and reconfiguring datacenter over impossible distances within a ridiculously short time frame. Prior experience as cable jockey, rack-n-stack grunt, console monkey, and/or log zombie a plus.

Moving a datacenter is a dirty job. Moving one halfway across the country in 48 hours -- that's a really dirty job. But that was the task facing Scott Wilson and his firm, Marathon Consulting, when one of its clients needed to close down its Chicago datacenter the day before Thanksgiving 2003 and open for business in New York the following Monday.

Wilson tried to persuade his financial services client to set up a duplicate center in New York; they could power down the Windy City operation, light up the Big Apple, then gradually move equipment as it was needed. No good, said the client -- too expensive. So at 4:30 p.m. on Wednesday, his people loaded roughly 80 machines into trucks and drove nonstop to New York.

"We tracked the trucks using GPS, so when they reached the Holland Tunnel, we went to the datacenter," says Wilson, managing director of the Brooklyn-based Marathon. "We spent the next 48 hours setting it up and getting operational. But we got it up."

Unplugging everything and cleaning out the muck that's collected over the years is bad, Wilson says. "Cables sit for years in half-baked air-conditioned rooms that are dusty and nasty."

But the worst part is putting Humpty Datacenter back together again. "Most datacenters aren't labeled correctly and have been put together by 10 different consultants and in-house employees who each have their own ways of doing things," Wilson says. "And recabling someone else's work is always fun."

Fortunately, migrating datacenters isn't something firms do very often. But when they do, it's an ordeal. IT pros resent having to do grunt work, but they also understand it's part of the job.

"On the other hand, moving 10 racks of servers from Chicago to New York in 48 hours at the end of the day feels amazing," says Wilson. "The gratification is definitely there."

Dirty IT job No. 1: Sludge systems architect

Seeking individuals with demonstrated ability to squeeze over, under, or between confined spaces to solve technical problems. Candidates should be prepared to work long hours for low pay under adverse conditions. Must not be allergic to sawdust, vermin, airborne pathogens, or sewage.

Sometimes dirty jobs are just that -- dirty. These days, technology goes everywhere: oil rigs, pulp mills, sewage plants, you name it. Somebody's gotta clean up the mess and keep the lights on.

"One of my early network projects was a network upgrade for a plywood mill," says Roberta J. Flinn, a senior IT architect for IBM Global Services' network practice in Beaverton, Ore. "We successfully found all but one of the switches to be upgraded. After a full day of searching and climbing around in the 'rafters,' we finally found the switch on a mezzanine above the planers. It was completely covered with about 6 inches of sawdust and still running."

But few IT gigs get earthier than Dan King's job as a process control engineer for a Texas sewage treatment facility in the mid-1990s.

"Among other things," King says, "I was responsible for crawling around the sludge dryer -- that's where the poo goes after it's extracted from the water -- trying to figure out how to program the computers to run the conveyors at speeds that would get the sludge dry enough so that it's not a sloppy muddy mess, yet not so dry and dusty that it would catch on fire."

A particularly smelly fire was the reason King was assigned to the project in the first place, he adds pungently.

To keep the "sludge" at the right consistency, King used an '80s-era programming language called CL, made by Honeywell Industrial Control Systems, to move the conveyor belts at precisely the right speed and send the right amount of electricity to the dryers. That was the easy part.

"Then I had to crawl around the belt and reach in with my glove to check the consistency of this muddy, slushy mess while watching the temperature."

After that formative experience, King went to grad school. He's now an SAP consultant and NetWeaver Integration specialist for CapGemini in Houston. He says even that job can get dirty sometimes, especially when he needs to convince clients to give his people access to the things they need to get their work done.

"Some days, I'm still up to my hips in poo, but it's bull poo," King says.

After writing this article, Contributing Editor Dan Tynan felt the need for a really long shower.