Unix Tip: Reading and Writing with File Descriptors

Send in your Unix questions today!


See additional Unix tips and tricks

Have you ever wanted to read a file one line at a time in a shell script and found the task to be a lot more trouble than you ever imagined? If you use a "for line in `cat file`" loop, for example, each pass through the loop is going to set your $line variable to, not the next line in the file, but the next word in the current line. One handy way around this problem is to assign a file descriptor to your input file and then use that file descriptor to read the file line by line. Here's how it works.

Reading from a File Descriptor



First, you can assign a file descriptor to an input file with this syntax:



$ exec 3< infile



where 3 is the file descriptor we have chosen to use and file is the name of the file.
Under most circumstances, we should avoid the use of file descriptors 0, 1 and 2, since these file descriptors are assigned to standard in, standard out and standard error. In the example shown, each time we read from infile, our position in the file will be incremented by one line.



To read from infile, we would then use a command like this:



$ read <&3 myline



The variable "$myline" will then contain the first line from the file. Repeat the read command and we will get the second line and so on. We might, therefore, use a script like that shown below to read and process each line in a file. Notice the comments inserted into the spot where we would process each line in some way in whatever way we want.
#!/bin/bash

if [ $# != 1 ]; then
    echo "Usage: $0 input-file"
    exit 1
else
    infile=$1
fi

# assign file descriptor 3 to input file
exec 3< $infile

# read til the end of the file
until [ $done ]
do
    read <&3 myline
    if [ $? != 0 ]; then
        done=1
        continue
    fi
    # process file data line-by-line
    # in here ...
    echo $myline
done

echo "The End"

This particular script is written in bash, but the exec command will work just the same in the Korn shell (/usr/bin/ksh) and Bourne shell (/usr/bin/sh). One difference between these shells with respect to user-defined file descriptors, however, is that the Korn shell (on Solaris anyway) only allows us to assign file descriptors 0 through 9 while bash and the Bourne shell work with file descriptors as high as 255.



If we issue a read command on a file descriptor without assigning a variable to hold the resultant data, we will find the data in $REPLY as shown in this interactive sequence of commands:

bash$ exec 3< infile
bash$ read <&3
bash$ echo $REPLY
One is the loneliest number that you'll ever do
bash$ read <&3
bash$ echo $REPLY
Two can be as bad as one
bash$ read <&3
bash$ echo $REPLY
It's the loneliest number since the number one

Notice how we are stepping through the lines in the file.



Writing to a File Descriptor



To assign a file descriptor to an output file, we use a very similar syntax to the exec command that we just used to read from a file:
$ exec 4> $outfile



In this command, file descriptor 4 is being assigned to $outfile. To write to this file, we would then use commands like these:



echo "File $file is unwritable" >&4
echo "Using /tmp/$file instead" >&4



When we examine the contents of $outfile, it will contain lines like these:
File data0506.log is unwritable

Using /tmp/data0506.log instead



Listing File Descriptors



On Solaris systems, we can list the file descriptors associated with a particular process with the pfiles command. If we were to run the two exec commands shown above on the command line, for example, we would see something like this when we run pfiles on the current shell:

$ pfiles $$
26036:  -bash
  Current rlimit: 32768 file descriptors
   0: S_IFCHR mode:0620 dev:102,0 ino:394018 uid:1111 gid:7 rdev:24,1
      O_RDWR|O_LARGEFILE
   1: S_IFCHR mode:0620 dev:102,0 ino:394018 uid:1111 gid:7 rdev:24,1
      O_RDWR|O_LARGEFILE
   2: S_IFCHR mode:0620 dev:102,0 ino:394018 uid:1111 gid:7 rdev:24,1
      O_RDWR|O_LARGEFILE
   3: S_IFREG mode:0644 dev:0,2 ino:476081584 uid:0 gid:1 size:1531
      O_RDONLY|O_LARGEFILE
   4: S_IFREG mode:0644 dev:0,2 ino:479974080 uid:1111 gid:10 size:0
      O_WRONLY|O_LARGEFILE
 255: S_IFCHR mode:0620 dev:102,0 ino:394018 uid:1111 gid:7 rdev:24,1
      O_RDWR|O_LARGEFILE FD_CLOEXEC

Notice file descriptors 3 and 4 in this output.



You will also see the file descriptors in lsof output. File descriptors 3 and 4 (shown as 3r and 4w) might look like this:

COMMAND     PID     USER   FD   TYPE     DEVICE SIZE/OFF      NODE NAME
bash       1394   bubba    3r  VREG        0,2     1531 476081584 /tmp (swap)
bash       1394   bubba    4w  VREG        0,2        0 479974080 /tmp (swap)

On some Unix systems, lsof will list the file names:

COMMAND    PID USER   FD   TYPE     DEVICE   SIZE/OFF     NODE NAME
bash      1394 bubba  3r  VREG       14,5        853  1352777 /tmp/infile
bash      1394 bubba  4w  VREG       14,5          0  1352778 /tmp/outfile

Reading and writing files using file descriptors can be very handy. We will look at some more applications of this Unix "trick" in next week's column.