Unix as a Second Language

Squeezing out the white space

Cleaning up text files to simplify subsequent processing is an easy job for Perl. The tool's easy use of regular expressions makes it a real champ at tasks like collapsing strings of blanks into a single character. Let's look at how this works.

Since Perl symbolizes any amount of white space (blanks, tabs and newlines) with the expression \s+, the substitution expression s/\s+/ / would squeeze any amount of white space down to a single blank. If you want to do this more than once on a line of text, all you have to do is add a "g" to the end of the line making it s/\s+/ /g. This line in a Perl script squeezes all the extra white space in the text contained in the default input:


s/\s+/ /g;


The following script will squeeze repeated blanks down to single characters and remove blanks from the beginning and end of the strings as well. The ^ (beginning of line) and $ (end of line) characters provide the secondary cleanup.


#!/usr/bin/perl -w
# squeeze out extra blanks

while (<>) {
s/\s+/ /g;
s/^\s//;
s/\s$//;
print;
}


Without a newline in the print statement, the output would look like this:


boson> echo " hello world " | ./squeeze
hello worldboson>


If this doesn't suit you, you can change this by adding another line to the script that appends a newline to your output. In fact, this is generally a good idea. Since the white space removed includes the newline characters, you might want to put them back.


#!/usr/bin/perl -w
# squeeze out extra blanks

while (<>) {
s/\s+/ /g;
s/^\s//;
s/\s$//;
s/$/\n/;
print;
}


If you pipe an entire file at the second version of the 'squeeze' script, you will still have separate lines in your file! In other words, the newlines will not be squeezed out as well.

You can do your white space squeezing on the command line if you prefer. If we look at the simplest Perl one-liner, we can start to imagine how to construct a command that would work for us. First, here's the simple one-liner:


perl -e 'print "Hello world";'


That works, of course, though is has no particular advantages over the simpler echo. In fact, echo would be more efficient. You could do something like this to remove the extraneous white space on the command line:


echo " Hello world " | perl -ne '$_=~ s/\s+/ /g; print;'


Here's the command in action:


boson> echo " Hello world " | perl -ne '$_=~ s/\s+/ /g; print;'
Hello world
boson> msg=`echo " Hello world " | perl -ne '$_=~ s/\s+/ /g; print;'`
boson> echo $msg
Hello world


I find the one-liner to be somewhat awkward and prefer the reusable script. It can, after all, be easily used on the command line or through another script.

Sign up for ITworld's Daily newsletter
Follow ITworld on Twitter @IT_world