Unix How To: Sed & Awk -- Still friendly after all these years

Even after decades of using Unix on thousands of systems, I find that it's still fun to discover various convolutions of sed and awk commands to perform command line wizardry. There's a lot more to each of these tools than those uses I make of these commands on a routine basis. Let's take a look at some one-liners you might not yet have tried.

One of my long-standing "tricks" for awk is using $NF to print the last field on every line. Since NF represents the number of fields on a line (e.g., 6), $NF represents the value of that last field (e.g., $6). Printing the last field of every line, therefore, might look like this:

$ awk '{print $NF}' myfile

or this:

$ awk -F: '{print $NF}' /etc/passwd

depending on whether the file is white space delimited or not.

You can get the length of every line in a text file by using awk's length command. For example, this command prints the length of every line in the /etc/motd file:

$ awk '{print length($0)}' /etc/motd

You can compute the square root of every fourth field in a file of numbers like so:

$ awk '{print sqrt($4)}' nums

You can display select lines from a chosen file.

$ awk '/nobody/ { print $0 }' /etc/passwd

You can generate an alphabetical listing of your users with an added sort command.

$ awk -F: '{ print $1 }' /etc/passwd | sort

You can count the lines in a file. In this example, we're using the END (only perform this after all lines have been processed) feature of awk to print the line number of the last line in the file.

$ awk 'END { print NR }' myfile

If you want to know how many words are in every line of a file, try this:

$ awk '{print NF}' myfile

You can add line numbers to every line in a file.

$ awk '{print NR, $0}' myfile

Sed, which I often find myself using only for those dead easy text substitutes such as sed "s/this/that/g" myfile also has some more interesting applications.

You can double space a file (appends a newline to every displayed line). Use 'G;G' (don't forget the quotes!) for triple spacing, 'G;G;G' for quadruple and so on.

$ sed G myfile

You can remove double spacing from a file.

$ sed 'n;d'

I've used sed to replace only the first instance of a string or every instance of a string in each line of text, but you can also use it to replace a specific instance -- for example, only the third or only the seventh instance.

$ sed 's/this/that/3'

You can also change multiple strings in a single sed command using semicolons like so:

$ sed 's/this/that/g;s/you/me/g;s/him/her/g' myfile

The sed and awk commands are far more versatile than I had remembered them being and could provide some useful aliases for you to toss into your startup files.

Infographic: Starting salaries for computer science grads
View Comments
You Might Like
Join the discussion
Be the first to comment on this article. Our Commenting Policies