Recovering from Proc

by Brian Hatch

May 7, 2002 —

I was sitting on my laptop looking at a movie [1] of my daughter Reegen
in her hula skirt on our trip to Hawaii this January. She and Allison,
the daughter of one of our friends, were playing "Ring around the
Rosey" and it was terribly cute.

Unfortunately, in a not-rare-enough multitasking mistake, I
accidentally deleted the movie I was watching! I was in the /movies
directory, and wanted to delete mvi_051*.avi and accidentally typed 'rm
mvi_051 *.avi', wiping out all my movies! No! It's not fair! I hadn't
made backups of the file yet and hadn't yet posted it to my Web site. A
beautiful moment lost. What was I to do? [2]

No problem, lsof and /proc to the rescue. I had lsof (List Open Files,
one of the must-haves in your UNIX administration and security
toolboxes) show me the open files for the mplayer [3] process:

$ lsof -c mplayer | grep avi
mplayer 10153 bri 10r REG 3,7 2545962 26708 /movies/hula.avi
(deleted)

This shows that mplayer (process 10153) has /movies/hula.avi opened on
file descriptor 10. (The rest of the data describe the device it's on,
size of file, etc....) The '(deleted)' at the end signifies that the
file has been deleted from disk. However, Linux files aren't actually
removed from the disk until all open file descriptors are closed and
all hard links to the data are removed. Thus, the file was still there;
I just couldn't get it by looking in the /movies directory because the
hula.avi name had been removed.

However it's still possible to get at the file. All I needed to do was:

$ cp /proc/10153/fd/10 /tmp/hula_recoved.avi

The /proc filesystem is not an actual directory on disk like /usr
or /home. Instead, /proc is a directory-based view of information the
kernel makes available to you. The programs ps or top, for example,
look in this directory for process ids and program names, and then
presents them in a pretty form. The files and directories in
the /proc/10153 directory refer to the process 10153 (mplayer). A quick
list shows us:

$ ls -F /proc/10153/fd
0@ 1@ 10@ 12@ 2@ 3@ 4@ 5@ 6@ 7@ 8@ 9@

$ ls -l /proc/10153/fd/10
lr-x------ 1 bri hle Apr 30 10:39 10 -> /tmp/reegen_hula.avi
(deleted)

The fd (file descriptor) directory has maps to the files open by the
program. So you can see that there are many file descriptors open (0,
1, 2... 10, and 12), /proc tries to show information in the most useful
UNIX-like way. Although it looks like /proc/10153/fd/10 is a symlink to
the file '/tmp/reegen_hula.avi (deleted)', when you try to copy this
file, it will give you the actual bits still on disk because the file
hasn't been permanently removed from the hard drive.

So, what does this have to do with security?

One common trick malicious hackers use is to open a file and
immediately delete it, such that the file is not visible on the machine
to tools like find/locate/etc. This also means that, if the machine is
rebooted, then the file disappears as well. Until the program stops,
the file is still completely usable to itself. It may be a temporary
storage space for lists of machines to compromise, copies of newly
downloaded attack scripts, or captured passwords to be sent back to the
attacker. Using deleted files is an easy method to avoid detection from
most administrators, and automatically removes any incriminating files
in the event an admin figures something is amiss and kills the process
or reboots the machine.

Next week, I'll show you a few other related /proc and lsof tidbits
that can be useful at preventing people from using this trick against
you. But for now, remember that if you delete something but still have
it open, you have an alternative to misery. In the event that the
deleted item is a cute movie of your daughter, you'll understand the
need.

NOTES
[1] http://www.hackinglinuxexposed.com/articles/hula.html
[2] If this drive were an ext filesystem, I may have been able to
recover all the files using e2undel
(http://e2undel.sourceforge.net/) or other similar tools.
Unfortunately I have most of my partitions formatted with
ReiserFS. Fortunately, the rest of the movies were already
backed up.
[3] Mplayer, a great Linux movie player, at
http://www.mplayerhq.hu/homepage/

ITworld