Last week, I introduced you to a bit of wizardry using the /proc
filesystem to recover a file deleted from the hard drive that was still
held open by a process on the machine. This week, I'll show you a few
other tricks you can use to investigate programs that are running on
your machine.

Again, the two tools we'll look at are lsof and the /proc filesystem.
Let's say we find a program called 'crackit' running on our system that
uses a significant bit of CPU. (Perhaps we noticed our machine was
sluggish and singled this process out by looking at 'top' output, for
example.)

First, we should always perform a simple ps of the program:

$ ps -f -p 24061
UID PID PPID C STIME TTY STAT TIME CMD
axe 24061 1087 99 09:36 pts/9 R 16:06 ./crackit 18277

A quick search for crackit on the hard drive turns up nothing:

$ locate crackit
$ find / -name crackit

Ok, so something is fishy here. Either the file has been deleted or the
command line has been altered.[1] Let's perform a full lsof of the
process to see what's going on:

$ lsof -p 24061
CMD PID USER FD TYPE DEVICE SIZE NODE NAME
john 24061 axe cwd DIR 3,5 4096 49430 /tmp/r (deleted)
john 24061 axe rtd DIR 3,7 1024 2 /
john 24061 axe txt REG 3,5 201840 49439 /tmp/r/john (deleted)
john 24061 axe mem REG 3,7 340771 40255 /lib/ld-2.1.3.so
john 24061 axe mem REG 3,7 4101836 40258 /lib/libc-2.1.3.so
john 24061 axe 0u CHR 136,9 29484 /dev/pts/9
john 24061 axe 1u CHR 136,9 29484 /dev/pts/9
john 24061 axe 2u CHR 136,9 29484 /dev/pts/9
john 24061 axe 3w REG 3,5 39 49446 /tmp/r/john.pot
(deleted)
john 24061 axe 4r CHR 5,0 29477 /dev/tty
john 24061 axe 5r REG 3,5 16157 49435 /tmp/r/password.lst
(deleted)
john 24061 axe 6uW REG 3,5 57 49447 /tmp/r/restore
(deleted)

ps duplicates a lot of this info, such as the tty (/dev/pts/9) that's in
use, the process id, and username. Hmmn, the command name here, however,
is 'john' instead of crackit. lsof retrieves the program name from a
different location in /proc, rather than from the command line of the
process, which can be faked. Looks like someone is trying to stealthily
run the damned-fine password cracker John the Ripper[2].

The lsof output shows that this program was being run in the /tmp/r
directory, which has been deleted since it was created (along with all
the files therein, such as the john binary itself, john.pot,
password.lst, and restore). We won't see the /tmp/r directory if we go
looking for it:

$ ls /tmp/r
ls: /tmp/r: No such file or directory

However, we could still recover those files. Let's copy them to
/root/mystery by accessing the filedescriptors themselves from /proc,
just like we did last week to recover our precious movie file:

# mkdir /root/mystery
# cd /root/mystery
# cp /proc/24061/fd/3 ./john.pot
# cp /proc/24061/fd/5 ./password.lst
# cp /proc/24061/fd/6 ./restore

Ok, we have a copy of the files in use by the password cracker, but
don't have a copy of the password cracker itself. Sure, we probably
don't need it, but it's nice to be able to see if the attacker has
improved this one by disassembling it or running it under a debugger.
However the binary that's running isn't open on a file descriptor, so we
can't copy from /proc/PID/fd. However it is available via a different
file in /proc:

# cp /proc/24061/exe ./john

This 'exe' file is a copy of the executable itself, so we've snagged it
too.

Ok, we now have a copy of the program and it's support files, despite
the attacker's attempts to deny them to us. Next week, we'll
investigate this a bit further, including an important answer to the
question 'So if the files are deleted, how will the cracker get access
to the decrypted passwords when they're cracked?' Don't forget, as soon
as the password cracker process is done, the /proc/PID directory that
belonged to it will disappear, taking those files with it....

NOTES

[1] Altering your command line is trivial, just edit your argv array and
ps/lsof/etc will all report whatever you want. To test this for
yourself, you could simply run this command line perl script, which
changes the $0 variable and then looks itself up in ps:

$ perl -e '$0 = "Fake command line"; system "ps -f $$"'
UID PID PPID C STIME TTY STAT TIME CMD
arioch 24403 24126 0 09:59 tty4 S 0:00 Fake command line

See how easy that was?

[2] http://www.openwall.com/john/

» posted by ITworld staff

ITworld

• Email
• Print
• Share
  • Slashdot
  • Digg
  • Stumble
  • Reddit
  • Newsvine