August 14, 2013, 1:50 PM —
Image credit: flickr/Sean Madden
Much of what sys admins learn — when they are not learning from trial and error — is knowledge passed on from other admins over coffee, online, in conference sessions and hallway tracks, and inside cubicle walls. With this "word of mouth" wisdom in mind, I asked seasoned sys admins to share some of the best advice they've received over the years, and here's what they said:
1. Always have good reference materials: Peter Galvin, Chief Solutions Architect for Pluribus Networks and former Sys Admin magazine columnist, recommended having technical reference materials nearby. You could start with a copy of the UNIX System Administration Handbook, or search the wonderful world of the Internet to find a plethora of practical sys admin articles and resources.
2. Don't be afraid to say, "I don't know.": Russ Harland, a senior systems administrator, said a former supervisor passed down this practical advice. "We had a challenging overnight cutover affecting thousands of systems, and it didn't go as smoothly as we would have liked. As a younger man at the time, I was being harshly self-critical about my perceived shortcomings and the things I didn't know." Russ's supervisor took him aside and offered a few encouraging words. "And then the kicker: 'Don't ever be afraid to admit you don't know something','" Russ said. "Nowadays, when I have to say 'I don't know,' I experience positive emotions, because I know I'm about to learn something." And really, who doesn't learn something new every day as a sys admin?
3. Expect small mistakes to be hugely irritating: During a big network cutover years ago, Harland and his team hit a few bumps on the road. "I remember being so drained and down when it was finally done because everything didn't go smoothly," he said. The experience made Harland question whether working in IT was the right fit for him. "Upon further reflection, I saw that the problems were not because of any shortcoming or negligence on my part — just a missing line in a config that made life hard for a couple of hours." Harland said the experience taught him to go easier on himself. "Our minds, because of the worlds we inhabit work-wise, insist on being completist in nature, with no detail unaccounted. That's just not life," he notes.
4. Plan for mistakes: Amy Rich has more than two decades of system administration experience under her belt, but she didn't start off as a sys admin. Rather, she switched her major from Aerospace Engineering to Computer Science her sophomore year. "I bought myself a Sun 3/60 and a copy of the 'Yellow Book,' the first edition of Evi Nemeth, Garth Snyder, and Scott Seebass's UNIX System Administration Handbook," Rich explained. "I convinced the college's head sys admin to let me put my workstation on the campus network and, with the help of that book and USENET, started experimenting with administering my own SunOS machine." Within a few weeks, Rich had a job in the computer science department as a work study student, reporting to a supervisor that had more faith in Rich's skills than she did. "I was terrified when he gave me the root password to all of the non-staff machines on the first day," Rich said. "But he said to me something along the lines of, 'Yeah, you're probably going to mess something up since you're just learning. The key thing to remember is to always make a backup first so you can roll back if things go wrong.'" Rich cited this as one of the best pieces of technical advice she's ever received. "And it's progressed from a simplistic 'make a copy of that file,' to 'tar up that directory,' to 'put that under revision control,' to 'always back up your data to tape/another machine, even your home machines,' to 'do real change management with written and tested recovery procedures'," she said, adding, "The seed he planted has allowed me to make mistakes and subsequently save my bacon on more occasions than I can possibly count over the past 20 years."
5. Respect Murphy's Law: "I think the most hard-won experience I have is a healthy respect for Murphy's Law," said hosted web help desk engineer Steven Klassen. He noted that there is only so much prevention you can apply and there will always be flaws in the best-laid plans. "When I'm asked how long a particular process is going to take to complete, I take the amount of time it would take me to do it in a vacuum and double or triple it. If I'm challenged on an estimate I explain that I'll do everything I can to do the work expediently but that I don't have control of every variable," Klassen said. Along these lines, sys admin Jussi Kekkonen learned from a colleague to plan for the unexpected. "Always have plan B and C, always ensure you can undo, etc.," Kekkonen said. "I can't even count how many times some hardware failed or a software update actually doesn't fix but breaks more."
6. Take breaks: "Probably the best advice I ever got was when I had hit a wall with a complex project I was working on and the senior sys admin told me to take a break and work on something smaller that I knew I could complete," said sys admin Elizabeth Krumbach Joseph. She took the advice. "Once I gained the confidence and thrill from completing the small project, I felt refreshed and much happier about returning to the complex one and was able to move forward," she said, adding, "It's a strategy I still take advantage of often."
7. Be paranoid: Sarah Baker, a senior Unix admin and production operations manager said that she learned "paranoia is a virtue" when she was working in a live production environment. "When any mis-step can be a job ender," she said. Early in her career, Baker learned the value of being paranoid and passes it on to other admins today. She noted that paranoia works to your advantage whether you are dealing with security or just a case of "fat fingers" on a keyboard. "I considered Evi [Nemeth] the tops in system admin/operations," Baker added. "I gave her book out to any new aspiring sys admin as a must read."
8. Be confident: IT director Jeremy Mayfield said that, when tackling issues in IT, think "I WIN." Mayfield recommended that admins remember that they are smarter than the issue at hand, and they have the skills and confidence to find a solution. "The company or organization is depending on you, and you must come through, therefore I WIN," he explained. Mayfield and his team use this mantra when working on routine, daily administration and when tackling larger projects. "This advice comes in handy every day," he said.
9. Do things right the first time: Confidence is great, but you should still take the time to do things right the first time, Mayfield explained. He said that when you hit budget restrictions, you should lay out the situation clearly so that your supervisors are aware that no budget means fewer options and IT can't survive on antiquated hardware. Sometimes, doing it right the first time means digging in your heels to get the necessary resources.
What's the best advice you ever got from another admin? What advice would you pass on? Let us know in the comments.