Unix How To: Peering at a Process with lsof

You have probably used lsof from time to time, probably when tracking down some sort of problem. But maybe you haven't tried all of its permutations or looked at it just to get a deeper understanding of how some particular process works. Let's look at one particular lsof command that can provide a lot of insight into a single process.

In this first example, we're going to look at the ypbind process with the command "lsof -c ypbind". The "c" specifies that we want to look at processes whose names start with "ypbind".

We see right away that ypbind is running as PID 198, making it likely it hasn't been restarted since the system was booted. We can also scan through the lines of output and see that it's using ports 32779 (UDP) and 32772 (TCP), both running IPv4.

bash-2.03# lsof -n -c ypbind
COMMAND PID USER   FD   TYPE        DEVICE SIZE/OFF   NODE NAME
ypbind  198 root  cwd   VDIR         136,0     2048      2 /
ypbind  198 root  txt   VREG         136,0    36256 115560 /usr/lib/netsvc/yp/ypbind
ypbind  198 root  txt   VREG         136,0  1158072 269602 /usr/lib/libc.so.1
ypbind  198 root  txt   VREG         136,0    13184 269407 / (/dev/dsk/c0t2d0s0)
ypbind  198 root  txt   VREG         136,0   919956 269353 /usr/lib/libnsl.so.1
ypbind  198 root  txt   VREG         136,0    17096 473904 /usr/platform/sun4u/lib/libc_psr.so.1
ypbind  198 root  txt   VREG         136,0    24968 269369 /usr/lib/libmp.so.2
ypbind  198 root  txt   VREG         136,0     5008 269399 /usr/lib/libdl.so.1
ypbind  198 root  txt   VREG         136,0   266140 269361 /usr/lib/ld.so.1
ypbind  198 root    0r  VCHR          13,2      0t0 864205 /devices/pseudo/mm@0:null
ypbind  198 root    1w  VCHR          13,2      0t0 864205 /devices/pseudo/mm@0:null
ypbind  198 root    2w  VCHR          13,2      0t0 864205 /devices/pseudo/mm@0:null
ypbind  198 root    3ww VREG         136,0        4 966677 /var/yp/binding/ypbind.pid
ypbind  198 root    4u  IPv4 0x3000015a4b0      0t0    UDP *:32779 (Idle)
ypbind  198 root    5ww VREG         136,0       15 966840 /var/yp/binding/xprt.udp.3
ypbind  198 root    6ww VREG         136,0       15 966841 /var/yp/binding/xprt.udp.2
ypbind  198 root    7ww VREG         136,0       15 966842 /var/yp/binding/xprt.udp.1
ypbind  198 root    8u  IPv4 0x3000015a030      0t0    TCP *:32772 (LISTEN)
ypbind  198 root    9ww VREG         136,0       14 966848 / (/dev/dsk/c0t2d0s0)
ypbind  198 root   10ww VREG         136,0       14 966849 / (/dev/dsk/c0t2d0s0)
ypbind  198 root   11ww VREG         136,0       14 966850 /var/yp/binding/xprt.tcp.1
ypbind  198 root   12u  VCHR        105,19      0t0 864213 /devices/pseudo/tl@0:ticots->timod->tl
ypbind  198 root   13ww VREG         136,0       14 966851 / (/dev/dsk/c0t2d0s0)
ypbind  198 root   14ww VREG         136,0       14 967065 / (/dev/dsk/c0t2d0s0)
ypbind  198 root   15ww VREG         136,0       14 967066 / (/dev/dsk/c0t2d0s0)
ypbind  198 root   16u  VCHR        105,20      0t0 864213 /devices/pseudo/tl@0:ticots->timod->tl
ypbind  198 root   17ww VREG         136,0       14 967067 /var/yp/binding/xprt.ticotsord.3
ypbind  198 root   18ww VREG         136,0       14 967077 /var/yp/binding/xprt.ticotsord.2
ypbind  198 root   19ww VREG         136,0       14 967079 /var/yp/binding/xprt.ticotsord.1
ypbind  198 root   20u  VCHR        105,21      0t0 864213 /devices/pseudo/tl@0:ticots->timod->tl
ypbind  198 root   21ww VREG         136,0       14 967080 /var/yp/binding/xprt.ticots.3
ypbind  198 root   22ww VREG         136,0      140  44801 / -- cache_binding

The FD and TYPE columns in this output require a little explanation. FD, as you likely suspect, stands for "file descriptor". But it contains values that we don't normally relate to file descriptors -- like "cwd" and "txt". It also shows "r", "u" and "w" characters after the numeric values.

The "cwd" descriptor displays the process' current working directory. The "txt" indicates that the associated files are program code and libraries (code and data). The 0-22 values represent what we normally think of as file descriptors. The extra characters show that the process has read (r), write (w) or read/write (u) access to each.

VREG (REG if you're not using Solaris) and VDIR (DIR) values in the TYPE column represent regular files and directories. CHR and BLK stand for character and block devices. You might also see UNIX, FIFO and IPV4 for Unix domain sockets, first-in-first-out queues and sockets.

The DEVICE and SIZE/OFF columns refer to the open files -- the disk, file size (or current position in the file) and inode (yes, / is inode 2). Then, of course, we have the file or device name under NAME.

To illustrate how lsof works, let's watch someone editing the /etc/passwd file. So let's try looking for vi.

bash-2.03# lsof -c vi
COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF   NODE NAME
vi      12357 root  cwd   VDIR  136,0     2048      2 /
vi      12357 root  txt   VREG  136,0   226656 896951 /usr/bin/ex
vi      12357 root  txt   VREG  136,0    17096 473904 /usr/platform/sun4u/lib/libc_psr.so.1
vi      12357 root  txt   VREG  136,0     5008 269399 /usr/lib/libdl.so.1
vi      12357 root  txt   VREG  136,0  1158072 269602 /usr/lib/libc.so.1
vi      12357 root  txt   VREG  136,0    42184 269355 /usr/lib/libgen.so.1
vi      12357 root  txt   VREG  136,0    16020 269344 /usr/lib/libcrypt_i.so.1
vi      12357 root  txt   VREG  136,0   466552 269345 /usr/lib/libcurses.so.1
vi      12357 root  txt   VREG  136,0    14556 269365 /usr/lib/libmapmalloc.so.1
vi      12357 root  txt   VREG  136,0   266140 269361 /usr/lib/ld.so.1
vi      12357 root    0u  VCHR   24,8   0t1731 864291 /devices/pseudo/pts@0:8->ttcompat->ldterm->ptem->pts
vi      12357 root    1u  VCHR   24,8   0t1731 864291 /devices/pseudo/pts@0:8->ttcompat->ldterm->ptem->pts
vi      12357 root    2u  VCHR   24,8   0t1731 864291 /devices/pseudo/pts@0:8->ttcompat->ldterm->ptem->pts
vi      12357 root    3u  VCHR  13,12      0t0 864206 /devices/pseudo/mm@0:zero
vi      12357 root    4u  VREG  136,0    24576 736000 /var/tmp/ExDOaOiy

OK, so we have some output, but no mention of vi under NAME. Why is this? With just a bit of checking, we confirm that vi and ex are the same executable (as shown by the following ls commands) and we do see ex listed in the second row.

# ls -li /usr/bin/ex
       384 -r-xr-xr-x   5 root     bin       227828 Jun 19  2002 /usr/bin/ex
# ls -li /usr/bin/vi
       384 -r-xr-xr-x   5 root     bin       227828 Jun 19  2002 /usr/bin/vi

We get a listing of the various libraries that vi is using -- seven of them in addition to ld.so.

We also see no mention of /etc/passwd in the files list. However, we do see that a temporary file has been opened in /var/tmp. Ah, yes, this is the directory that vi uses to store files that it is working on. We can check the /var/tmp/directory and verify the file name and size.

bash-2.03# ls -l /var/tmp/Ex*
total 26814
-rw-------   1 root     other      24576 Jan 26 16:49 ExDOaOiy

And if, out of some dweebish curiosity, we want to verify that this "Ex" file is indeed the /etc/passwd file that is being edited, we can steal a look at it. Just be prepared for it to look a little bit different in the form of a vi temporary file.

bash-2.03# more /var/tmp/ExDOaOiy
K_c{/etc/passwd

ÿÿ

root:x:0:1:Super-User:/:/bin/sh:/bin/bashdaemon:x:1:1::/:bin:x:2:2::/usr/bin:sys
...

Yes, that's certainly the /etc/passwd file.

The lsof command can provide an interesting view of how a particular process works as well as what it is working on and with.

Insider: How the basic tech behind the Internet works
Join the discussion
Be the first to comment on this article. Our Commenting Policies