Unix: Rooting out the kits

Rootkits are both tricky and stealthy, but there are still some things that you can do if you suspect that one of your Linux system has been infected. After all, a rootkit is going to be doing something if it's to be of any value to the miscreants that deployed it. In addition, its authors will have had a hard time trying to engineer their tools to avoid everything that detection tools are going to throw at it to identify and remove it.

The bad news is that detecting rootkits takes far more insight than noticing and identifying your typical virus. Many are designed to resemble device drivers so that it's possible for them to run at a more privileged level in the operating system. Rootkits often replace a keyboard or network driver, for example. The way that modern operating systems are broken into distinct privilege "layers" and numerous modules, loaded when needed and each of which manages a distinct function within the OS, makes this possible.

Sometimes root kits will replace commands such as netstat, du, find, ifconfig, netd, killall and lsof while they will just provide support for other malware -- allowing it to run undetected or providing access to the system through backdoors. The flexibility and modularity of operating systems is, thus, also something of a "weak link" as far as security is concerned.

When you suspect a rootkit has been installed on a system, the first thing you need to decide is what the first step ought to be. Some will say that you should immediately detach it from your network, isolating it for further analysis. Others will say that you may lose valuable insights into what the rootkit is doing if you move too quickly. Besides, depending on the role the system is playing, pulling it off the network could have drastic implications if provides a critical service. On a well designed network, your critical services will be set up in such a way that you can roll them over to another system.

If your aim is to learn as much as you can about the rootkit, rebooting the system might be a bad idea. The rootkit might be one that is confined to memory and your evidence may be gone if you reboot too soon. In any case, this – how to proceed when a rootkit is suspected -- kind of decision is one that should be made long before you have to act.

You should consider detaching from your network and, at some point, shutting down the system and booting in single user mode. The key question is what's more important -- figuring out what happened or getting the system up and working again. If you must get it online again as quickly as possible, are you prepared to make an image of the infected system for analysis? If you can, that image might provide you with important insights after the fact.

It's a good to have a rescue CD or DVD on hand so that you can look at an infected system (or a potentially infected system) without depending on tools or commands that are installed on the system. When a rootkit is likely, all of them are suspect.

Some of the tools and commands you should have on that rescue CD or DVD include chkrootkit -- an open source application for checking for the presence of rootkits and mk5sum. On rpm-based systems, you will also find that the rpm command itself can prove very useful in detecting file changes that can tell you a lot about what files might be involved in your rootkit.

You can get chkrootkit from http://www.chkrootkit.org. It will:

• check system binaries for modification
• check if the network interface is in promiscuous mode
• analyze lastlog, wtmp and utmp files for potential deletions (also wtmpx on Solaris systems)
• look for Loadable kernel module trojans

You can use the find command to locate files that have the immutable (sticky) bit set -- something that some rootkits will do to keep the hacked files from being removed. You can also find files with the suid and sgid bits, but this might be less valuable than you might first think since rootkits will be running with root privilege and, thus, don't require this kind of support.

$ find / -type d -perm -1000 -exec ls -ld {} \;

You can also do some clever things with the rpm command. The rpm -V or rpm --verify command will show file and metadata changes that have occurred since installation. These commands look at checksums as well as permissions and file sizes Use rpm -V packagename to verify a single application. Use the -Va options to check all installed packages.

You can expect to see a sequence of dots like S.5....T followed by the name of a changed file. When you do, you'll know what has changed. Each of the letters between the dots has a particular meaning.

letter	meaning (i.e., what has changed)
======	================================
S	file size
M	mode
5	MD5 sum
D	device major/minor numbers
L	readLink path
U	user ownership
G	group ownership
T	mtime

If you get no output with rpm -V, you can add a lowercase v to see a list of the files being checked. They should be preceded by a string of dots showing that none of the changes described above were detected.

There are many free and commercial tools for detecting and removing rootkits. And any rootkit designer is going to be using them too. The best tool likely depends on the particular rootkit and we are likely to continue to see the game of one-upmanship (rootkit authors trying to outsmart detectors and vice versa) for many years to come.

roots.jpg