Security consulting not for the faint of heart

March 22, 2001 —

Bob Toxen

Bob Toxen's new book, Real World Linux Security: Intrusion Prevention, Detection, and Recovery, appeared on store shelves late last year. Toxen, now the president and CTO of Fly-By-Day Consulting, sports a colorful professional résumé with an abundance of highlights: he's the creator of the Sunset Computer, one of the 162 recognized developers of Berkeley Unix, one of the four developers who did the initial port of Unix to the Silicon Graphics hardware, and the software architect of the Netgear ND508 and ND520, as well as of the Kennedy Space Center PC space shuttle payload document network.

We held a discussion with Toxen in ITworld.com's Interviews forum and found out some of the perks and problems of the security consulting business. You can read the complete conversation there; what follows is an abridged version.

ITworld.com: How does the security business work for you, Bob? You obviously have plenty of knowledge that organizations need, but it can still be a challenge working out contractual relations so that everyone comes out a winner. How is security consulting different from other kinds of IT administration, development, or management? Should folks who want to work in the area prefer in-house employment to free agency, or vice versa?

Bob Toxen: The security business works well for me. Certainly, for me as for any consultant, contractual issues are a problem. Even with "ordinary" consulting, I've had to refuse unreasonable contracts and have lost the occasional project because of it. One company, with its corporate attorney still "wet behind the ears", insisted that I sign a new contract for an ongoing project that required me to indemnify them against any lawsuit by any customer over any software that I wrote or modified for them. This means that I would be financially liable even for suits without merit in code that subsequently had been altered by others or that had existing bugs before I touched it. Since the code was an online banking system being sold to large banks, I told the company "no" after I saw the contract. Fortunately, one of the company's founders overruled this lawyer. This has happened a few times since. I tell clients that I do a high-quality job but I'm not an insurance company.

Security consulting is not that different from the programming and system administration consulting that I've been doing for 10 years. Half of my sysadmin customers call me in a desperate panic because their systems are dead and they don't know what to do and there's valuable data on them or they're involved in online commerce. One recent call was received at 4:30 a.m. my time from another continent. (My office phone also rings at home to cover such emergencies.)

My advice to both types of clients is: please prepare for problems in advance. I can help you recover from a rm -rf / or security breach far faster and with far less overall cost if these everyday occurrences are prepared for ahead of time.

The difference between security consulting and ordinary consulting is that for ordinary jobs, if one leaves something out or does not know something, it can be added later or looked up somewhere else. But if I fail to close a hole in a security client's system, that system gets broken into.

Regarding those interested in being security specialists: all but the biggest entities will want a sysadmin who also has good security knowledge; my book certainly is good preparation. There are some good security courses but there is a lot of hype too. The largest entities have separate security departments and these might be good for those that want full-time security work.

Being a free agent consultant is hard work. Anyone can throw $700 at an attorney to create a corporation and have business cards printed up. Finding the work and -- more importantly -- not being burned on bad deals takes business sense and experience. Consulting also requires sacrifice to do right. I was about to sit down to dinner last night when a client called about a crashed system. I didn't get home until almost midnight. My client's people also worked into the night, but they could take turns being on call. This was a hardware problem that required some reconfiguration to work around. I once interrupted a vacation when a client did the above-mentioned rm -rf / and I had to spend days recovering his data from the free list; I was able to recover about 90 percent of it. He didn't back it up despite the fact that I had urged him to do so; he even owned a tape drive.

Like every small business owner, I also spend far too much time complying with government requirements and dealing with IRS auditors who don't even understand the regulations that they're supposed to enforce and who won't listen to reason. Tip: If the IRS questions something, don't even bother writing a certified letter; they'll just ignore it and threaten to empty your bank account. Go visit the local office; an appointment is not necessary and the phones may be busy anyway. If that doesn't get results, write a letter to your congressional representative. The IRS will flag your file as "under congressional investigation" and your case will be handled by a special department that knows it's being watched by someone powerful. Ask that they waive penalties because "you really did try to comply with regulations."

ITworld.com: I'm starting to think that we're in for a round of legislation, litigation, and certification as the maturing fields of administration and security sort out "best practices" and their tolerance for anything less. Is this affecting you, yet? Do you have any certifications, or opinions about them?

ITworld.com: Where there is money and publicity, lawyers and legislators will not be far behind. Good laws could go a long way toward protecting people. Laws and proposed laws that discourage encryption by "good guys" and do not distinguish between hacking and unauthorized use of a bank's computer to play solitaire do not help. While I think that breaking into someone else's computer should be illegal, I find it interesting that Mitnick got a more severe sentence than the ones handed down to other white-collar criminals -- or even most murderers.

I do not have any security certifications. While certifications may help eliminate "know-nothings," they also can be harmful. In many cases, the certifications are created by for-profit companies more interested in making money than helping security. Such a company was the only entity to refuse my request to use copyrighted material in my book out of dozens of requests that I made.

Good certifications can ensure a minimal level of competence, at the time that the test was given, but often it is an excuse for a manager to not make the effort to thoroughly interview potential vendors. I offer some tips on this in my book in Appendix B.12: "Consultants: The Good, the Bad, and the Slick".

ITworld.com: So, have you been in the situation yet where you've had to say, "It's a good plan, except you need to triple your budget," because your client has entirely left out security considerations? I expect a lot of projects to be cancelled once managers realize their liability under the child privacy and medical privacy laws in the United States.

Bob Toxen: As a consultant, I must walk a tightrope. Unless a client specifically asks me about budgets, I will not consider it to be my place to comment on them. When asked, I'll provide estimates of the expenditures needed to achieve various security goals. On the other hand, if I am aware of a problem and do not make it known to a client, I risk a lawsuit. Usually I point out the problems that I am aware of, offer to look for more if they like, and recommend that they buy the book. If they follow everything in the book, then they have done essentially all of the right things.

The recent US medical privacy laws are so severe that I understand that many hospitals plan to outsource their MIS work in order to outsource their liability. At the same time, each person's insurance company is allowed to force him to fully disclose all of his medical data and then is allowed to put it in a database freely accessible by every other insurance company. Read your policy! Since I'm a pilot, the US Federal Aviation Administration requires me to reveal every detail of my medical history under penalty of prison regardless of whether a condition affects my ability to pilot an airplane.

At least the laws discourage anything embarrassing from making the papers, though it usually comes out anyway. Because of the laws, I won't get involved in medical data unless I'm indemnified.

ITworld.com: How much work do you do overseas? Do you have any useful generalizations about how security engineering differs in the United States and elsewhere? (I like the book's coverage of laptops, by the way.)

Bob Toxen: I've done work directly in three European countries and I have had to turn down trips to two more due to conflicts with previous engagements, one being my mostly annual Halloween Party -- a very big event. I have worked with people from a number of other countries and continents. I have not found any differences. Engineering seems to be universal.

Speaking of laptops and international travel, many laptops are stolen at airport security checkpoints; I warn about this in the book and tell how to avoid it.

Additional resources

ITworld.com