Our Continuing /proc and lsof Investigation

by Brian Hatch

May 20, 2002 —

Let's take a quick look at the files available for our running mystery
process (which we've already determined is likely the John the Ripper
password cracker) to see what we can learn. The process id was 24061, so
let's look at the files therein:

$ cd /proc/24061
$ ls
-r--r--r-- 1 axe axe 0 May 13 13:42 cmdline
lrwx------ 1 axe axe 0 May 13 13:42 cwd ->
/tmp/r (deleted)
-r-------- 1 axe axe 0 May 13 13:42 environ
lrwx------ 1 axe axe 0 May 13 13:42 exe ->
/tmp/r/john (deleted)
dr-x------ 2 axe axe 0 May 13 13:42 fd
pr--r--r-- 1 axe axe 0 May 13 13:42 maps
-rw------- 1 axe axe 0 May 13 13:42 mem
lrwx------ 1 axe axe 0 May 13 13:42 root -> /
-r--r--r-- 1 axe axe 0 May 13 13:42 stat
-r--r--r-- 1 axe axe 0 May 13 13:42 statm
-r--r--r-- 1 axe axe 0 May 13 13:42 status

Some of these bits we've already seen interpreted by ps or lsof, such as
the cwd (current working directory) and the files inside fd (the open
file descriptors.) The cmdline file contains the command line that was
used to start the program -- or whatever the program altered it to.
These come straight out of the argv array, for you C programmers. Each
argument is separated by a null, which means that it looks all jumbled
together if you simply 'cat' it. Say I launch 'find / -name \*.mp3 -exec
chmod a+r {} \;' in a window and look at it's cmdline entry

$ cat cmdline
find/-name*.mp3-execchmoda+r{};$

That '$' at the end is our prompt. Pretty damned unreadable. An easy way
to parse out the command line usably would be to use this short command
line perl script:

$ perl -pe 's/\0|$/\n/g' cmdline
find
/
-name
*.mp3
-exec
chmod
a+r
{}
;

The environ file has a complete listing of all environment variables
that are set by the program in question. You could get this list using
'ps eww' for example:

$ ps eww -p 24061
PID TTY STAT TIME COMMAND
24061 tty3 R 4:52 ./john smarmy _=./john
PATH=/usr/local/bin:

:/bin:/usr/bin:/usr/X11R6/bin:/opt/bin:/opt/sbin:/sbin:/usr/sbin:/etc:/usr/etc:
/var/yp:/usr/lib:/usr/local/sbin:/usr/games:/home/bri/bin:.
SHELL=/bin/ksh
EDITOR=vi HOSTNAM....

But that's pretty darn ugly. Instead you could parse the environ file
(it also uses nulls between entries) just like we did the cmdline file:

$ perl -pe 's/\0|$/\n/g' environ | sort | head -6
DISPLAY=localhost:10.0
EDITOR=emacs
HOME=/home/axe
HOSTNAME=rooted
HOSTTYPE=i386
LANG=en_US

It would seem that the attacker didn't cleanse his environment at all,
and we may have some useful info in here.

The maps file shows memory mappings in use by the program:

$ head -2 maps
40000000-40013000 r-xp 00000000 03:07 40255 /lib/ld-2.1.3.so
40013000-40014000 rw-p 00012000 03:07 40255 /lib/ld-2.1.3.so

These were already visible in lsof output, but you get more information
about them through the maps file, if you are savvy enough to use it.

The mem file is an interface directly into the process' memory, meaning
you could read (or change) the real bytes that make up the program and
it's data. Stat, statm, and status all contain (sometimes repetitive)
status information about the process. See the proc(5) man page for a
detailed description. The status file is the most humanly readable
version:

$ head -6 status
Name: john
Stat: R (running)
Pid: 24061
PPid: 1087
Uid: 1020 1020 1020 1020
Gid: 1100 1100 1100 1100

Note that the 'name' here is the original program name, and cannot be
modified by changing argv. This is the name that lsof uses.

The two files cwd and root are symbolic links that point to the process'
current working directory and root, respectively. Using the chroot
system call, you can restrict a program to live entirely within a
specific directory, unable to access files outside of the chroot area.
This is common with security-conscious network services. In our case the
process is running from a deleted /tmp/r directory, with no special
chrooting.

That pretty much covers the important uses of /proc/PID meta-files.
However we're still stuck with a question about our mystery password
cracking process: since it's running with all the files it uses already
deleted, how will the attacker be able to recover these files when the
password cracker is finished? It may be that he's modified John to copy
the decrypted passwords somewhere else when done, or send them to him in
email, or other possibilities. Since the cracker is only running the one
process, this seems the most likely answer. However it's worth checking
if any other processes open those deleted file descriptors:

$ lsof | grep deleted
save 24070 samson 4r REG 3,5 39 49446 /tmp/r/john.pot
(deleted)
save 24070 samson 5r REG 3,5 57 49447 /tmp/r/restore
(deleted)
save 24070 samson 6r REG 3,5 16157 49435
/tmp/r/password.lst (deleted)

Hmmn...seems the user samson has a program called 'save' with access to
those same deleted files. Even when John quits, those files will be
accessible to the cracker to copy or send as desired. Perhaps the
cracker thought that using a different user would fool us. Or perhaps
samson is the real mischievous entity, and is trying to get 'axe' in
trouble, but reap the rewards of the password cracking. Regardless, now
is the time to investigate these processes as well.

ITworld