Purging the process, Part 2

Unix Insider –

Last month I covered several basics, such as input redirection:

<font face="Courier">
$ grep "hello" <hello.txt
say hello.
</font>

Output redirection:

<font face="Courier">
$ grep "hello" >junk.txt
Now is the time
for every good person to
say hello.
(type control-D here)
$ cat junk.txt
say hello.
$
</font>

Input and output redirection, and the use of input files on the command line instead of redirected input:

<font face="Courier">
$ grep "Now" <hello.txt >junk.txt
$ grep "Now" hello.txt >junk.txt
</font>

Appending additional data to a file using an output redirection:

<font face="Courier">
$ echo "Now is the time" >hello.txt
$ echo "for every good person to" >>hello.txt
$ echo "say hello." >>hello.txt
$ cat hello.txt
Now is the time
for every good person to
say hello.
$
</font>

Redirecting standard output and standard error, and redirecting standard error to the

<font face="Courier">/dev/null</font>
byte wastebasket:

<font face="Courier">
$ find / -name *.txt -exec ls -l {} \; 2>/dev/null >textfiles
$
</font>

Basic pipes:

<font face="Courier">
$ grep "hello" < hello.txt | sed -e "s/hello/bye/" > result.txt
$( grep "hello" | sed -e "s/hello/bye/" ) < hello.txt > result.txt
$
</font>

I also stated that redirecting output to an existing file would delete the file and create a new version of it. In the following example, the fourth line causes

<font face="Courier">hello.txt</font>
to be overwritten with a new version of the file containing only a single line,
<font face="Courier">bye</font>
.

<font face="Courier">
$ echo "hello" >hello.txt
$ cat hello.txt
hello
$ echo "bye" >hello.txt
$ cat hello.txt
bye
</font>

You can set the

<font face="Courier">noclobber</font>
option to prevent redirected files from automatically overwriting their predecessors. In the following example, the option causes an error message at line six when the user tries to overwrite the
<font face="Courier">hello.txt</font>
file.

<font face="Courier">
$ set noclobber
$ echo "hello" >hello.txt
$ cat hello.txt
hello
$ echo "bye" >hello.txt
File "hello.txt" already exists
$ cat hello.txt
hello
unset noclobber
</font>

If

<font face="Courier">noclobber</font>
is set, you can force a redirection to clobber any pre-existing file by using the
<font face="Courier">>|</font>
redirection operator. This operator looks like a redirection to a pipe, but it's actually just a force redirect to override the
<font face="Courier">noclobber</font>
option. In the following example the forced redirection operator prevents any error messages.

<font face="Courier">
$ set noclobber
$ echo "hello" >|hello.txt
$ cat hello.txt
hello
$ echo "bye" >|hello.txt
$ cat hello.txt
bye
unset noclobber
</font>

Combining standard output and standard error

Redirection is frequently used for jobs that run for a long period of time, or for jobs that produce a lot of output. For such jobs, redirection can capture the results in a file. When this is done, it's also necessary to capture any output errors. Remember that if you redirect standard output but not standard error, output will go to a file and error messages will still go to your screen. The following

<font face="Courier">find</font>
command will save the results to
<font face="Courier">found.txt</font>
, although errors still appear on the screen.

<font face="Courier">
$ find / -name *.txt -exec ls -l {} \; >found.txt
find: /some/directory: Permission denied
find: /another/one: Permission denied
$
</font>

The redirection operator is actually a number followed by the redirection symbol, as in the following example. If number is omitted, 1 is the default.

<font face="Courier">
$ find / -name *.txt -exec ls -l {} \; 1>found.txt
$
</font>

The following commands are equivalent:

<font face="Courier">
$ find / -name *.txt -exec ls -l {} \; 1>found.txt
$ find / -name *.txt -exec ls -l {} \; >found.txt
$
</font>

Unix utilities open three files automatically when a program starts up. These files are given file descriptor numbers inside the program -- 0, 1, and 2 -- but they're more commonly known as stdin (standard input -- file descriptor 0), stdout (standard output -- file descriptor 1), and stderr (standard error -- file descriptor 2). When the program starts, default assignments for these files are made to

<font face="Courier">/dev/tty</font>
, which is the device name for your terminal. The stdin file is assigned to the keyboard of your terminal, while stdout and stderr are assigned to the screen of your terminal. The output redirection operator defaults to 1; thus
<font face="Courier">></font>
and
<font face="Courier">1></font>
are equivalent. The input redirection operators
<font face="Courier"><</font>
and
<font face="Courier"><0</font>
are equivalent. Redirecting standard error, file descriptor 2, requires that its number be explicitly included in the redirection symbol.

The following examples use

<font face="Courier">1></font>
to redirect standard output because it helps clarify how the redirection works. When reviewing these examples remember that
<font face="Courier">></font>
and
<font face="Courier">1></font>
are the same.

One method of handling the logging problem would be to create separate logs for each of the outputs, as in the following example.

<font face="Courier">
$ find / -name *.txt -exec ls -l {} \; 1>found.txt 2>errors.txt
$
</font>

It is also possible to redirect an output by attaching it to an already open redirection using the

<font face="Courier">>&</font>
redirection operator. In the following example, the standard output of
<font face="Courier">find</font>
is redirected to the file
<font face="Courier">result.txt</font>
. The
<font face="Courier">2>&1</font>
redirection command instructs the shell to attach the output from standard error (2) to the output of standard output (1). Now both standard output and standard error are sent to
<font face="Courier">result.txt</font>
.

<font face="Courier">
$ find / -name *.txt -exec ls -l {} \; 1>result.txt 2>&1
$
</font>

The order of redirection is important. In the following example, the output of file descriptor 2 (standard error) is attached to file descriptor 1. At this point, standard output is still attached to the terminal, so standard error is sent to the terminal. The next redirection sends standard output to

<font face="Courier">result.txt</font>
. This redirection doesn't drag file descriptor 2 along with it, so standard error is left pointing to the terminal device.

<font face="Courier">
$ find / -name *.txt -exec ls -l {} \; 2>&1 1>result.txt
find: /some/directory: Permission denied
find: /another/one: Permission denied
$
</font>

Input redirection from here documents

Perhaps one of the most useful forms of redirection is redirecting input from a

<font face="Courier">here</font>
document. A shell script can be written that executes a command and serves all input to the command. This is frequently used for a command that is normally run interactively. As an extreme example, I will show you how to do this with the editor vi. I am using vi for two reasons: first, it's interactive, and second, you're probably fairly familiar with it already and so will have a better understanding of what the script's doing. Normally, hands-off editing is done with the
<font face="Courier">sed</font>
command.

First, create a text file with several

<font face="Courier">hello</font>
strings in it, as in the following example, then name it
<font face="Courier">hello.txt</font>
.

<font face="Courier">
sample hello.txt
hello world
hello broadway
hello dolly
</font>

Create a file named

<font face="Courier">here.sh</font>
that contains the lines in the example below. The second line starts the vi editor on the
<font face="Courier">hello.txt</font>
file and the
<font face="Courier"><<END-OF-INPUT</font>
option states that vi will run taking its input from this current file,
<font face="Courier">here.sh</font>
, reading in a line at a time until a single line containing
<font face="Courier">END-OF-INPUT</font>
is read in. The subsequent lines are vi commands to globally search for
<font face="Courier">hello</font>
, replace each instance of it with
<font face="Courier">bye</font>
, write the file back out, then quit. The next line is the
<font face="Courier">END-OF-INPUT</font>
line and final echo statement to indicate that the editing is complete.

<font face="Courier">
# here.sh - sample here document
vi hello.txt <<END-OF-INPUT
:g/hello/s//bye/g
:w
:q!
END-OF-INPUT
echo "Editing complete"
</font>

Change the mode on the file to make it executable:

<font face="Courier">
$ chmod a+x here.sh
</font>

When you execute the

<font face="Courier">here.sh</font>
script, you may receive a warning from vi that it's not running in interactive mode. Next, the actual editing takes place; afterwards, you can
<font face="Courier">cat</font>
out the
<font face="Courier">hello.txt</font>
file and see your handiwork.

<font face="Courier">
$ ./here.sh
Vim: Warning: Input is not from a terminal
Editing complete
$ cat hello.txt
sample bye.txt
bye world
bye broadway
bye dolly
</font>

If you really want to suppress the vi warning, redirect the error to the

<font face="Courier">/dev/null</font>
device, as in the following version of
<font face="Courier">here.sh</font>
:

<font face="Courier">
# here.sh - sample here document
vi hello.txt 2>/dev/null <<END-OF-INPUT
:g/hello/s//bye/g
:w
:q!
END-OF-INPUT
echo "Editing complete"
</font>

<font face="Courier">here</font>
documents frequently appear as small pieces of larger scripts. In order to make the
<font face="Courier">here</font>
portion stand out, it's helpful to indent that section of the shell. Using a minus (
<font face="Courier">-</font>
) in front of the end-of-input marker eats the white spaces at the beginning of a line and prevents them from being passed on to the program. The following is an example:

<font face="Courier">
# here.sh - sample here document
vi hello.txt 2>/dev/null <<-STOP-HERE
:g/hello/s//bye/g
:w
:q!
STOP-HERE
echo "Editing complete"
</font>

Because it's an interactive program, the

<font face="Courier">ftp</font>
utility is a common candidate for
<font face="Courier">here</font>
document status. The following example starts
<font face="Courier">ftp</font>
and redirects standard output and standard error to
<font face="Courier">xfr.log</font>
. The process logs in to a remote system named
<font face="Courier">nj_system</font>
, switches to binary transfer mode, creates two directories, transfers a file named
<font face="Courier">newstuff.a</font>
to the remote system, and signs out again. Using a
<font face="Courier">here</font>
document makes it possible to execute
<font face="Courier">ftp</font>
through a shell script while seeing what the script is doing. The second example below is another method of doing this, but it involves a separate file with the
<font face="Courier">ftp</font>
commands.

<font face="Courier">
# xfr.sh - Transfers to a remote system
district=nj
ftplog=xfr.log
insbase=/usr/installations
insdir=$insbase/new
inskit=newstuff.a
echo "Transferring to" $district
ftp 1>>$ftplog 2>&1 $district"_system" <<-ALL-DONE
        user mo ddd789
        binary
        mkdir $insbase
        chmod 777 $insbase
        mkdir "$insdir"
        chmod 777 $insdir
        put $inskit $insdir/$inskit
        chmod 777 $insdir/$inskit
        bye
ALL-DONE
echo "Transfer to" $district "complete."
</font>

The first file would have to contain nothing but the commands for

<font face="Courier">ftp</font>
, and couldn't take advantage of script variables. Here's a sample input for
<font face="Courier">ftp</font>
:

<font face="Courier">
user mo ddd789
binary
mkdir /usr/installations
chmod 777 /usr/installations
mkdir /usr/installations/new
chmod 777 /usr/installations/new
put newstuff.a /usr/installations/new/newstuff.a
chmod 777 /usr/installations/new /newstuff.a
bye

# xfr.sh - Transfers to a remote system
district=nj
ftplog=xfr.log
echo "Transferring to" $district
ftp 1>>$ftplog 2>&1 $district"_system" <ftp_commands
echo "Transfer to" $district "complete."
</font>

In our next installment, I'll cover Unix system and global variables. What are they and how do you use them? I have been meaning to do this one for a while, and now seems like a good time.

Top 10 Hot Internet of Things Startups
Join the discussion
Be the first to comment on this article. Our Commenting Policies