Operating systems

Squeezing out the white space

October 29, 2008, 10:22 AM — 

Cleaning up text files to simplify subsequent processing is an easy job for Perl. The tool's easy use of regular expressions makes it a real champ at tasks like collapsing strings of blanks into a single character. Let's look at how this works.

Since Perl symbolizes any amount of white space (blanks, tabs and newlines) with the expression \s+, the substitution expression s/\s+/ / would squeeze any amount of white space down to a single blank. If you want to do this more than once on a line of text, all you have to do is add a "g" to the end of the line making it s/\s+/ /g. This line in a Perl script squeezes all the extra white space in the text contained in the default input:


s/\s+/ /g;

The following script will squeeze repeated blanks down to single characters and remove blanks from the beginning and end of the strings as well. The ^ (beginning of line) and $ (end of line) characters provide the secondary cleanup.


#!/usr/bin/perl -w
# squeeze out extra blanks

while (<>) {
s/\s+/ /g;
s/^\s//;
s/\s$//;
print;
}

Without a newline in the print statement, the output would look like this:


boson> echo " hello world " | ./squeeze
hello worldboson>

If this doesn't suit you, you can change this by adding another line to the script that appends a newline to your output. In fact, this is generally a good idea. Since the white space removed includes the newline characters, you might want to put them back.


#!/usr/bin/perl -w
# squeeze out extra blanks

while (<>) {
s/\s+/ /g;
s/^\s//;
s/\s$//;
s/$/\n/;
print;
}

If you pipe an entire file at the second version of the 'squeeze' script, you will still have separate lines in your file! In other words, the newlines will not be squeezed out as well.

You can do your white space squeezing on the command line if you prefer. If we look at the simplest Perl one-liner, we can start to imagine how to construct a command that would work for us. First, here's the simple one-liner:


perl -e 'print "Hello world";'

That works, of course, though is has no particular advantages over the simpler echo. In fact, echo would be more efficient. You could do something like this to remove the extraneous white space on the command line:


echo " Hello world " | perl -ne '$_=~ s/\s+/ /g; print;'

Here's the command in action:


boson> echo " Hello world " | perl -ne '$_=~ s/\s+/ /g; print;'
Hello world
boson> msg=`echo " Hello world " | perl -ne '$_=~ s/\s+/ /g; print;'`
boson> echo $msg
Hello world

I find the one-liner to be somewhat awkward and prefer the reusable script. It can, after all, be easily used on the command line or through another script.

Sign up for ITworld's Daily newsletter
Follow ITworld on Twitter @IT_world

I like it!
Comments

Squeeze play somewhat simplified?

Just a dumb question, but wouldn't:

tr -s ' ' < fat.file > thin.file

accomplish the same thing w/o hurting the newline character? It wouldn't do the beginning/end of line single blanks, but it would squeeze 2 or more concurrent blanks into one.
| reply

syntax problem?

Should the one liner shown as:

echo " Hello world " | perl -e '$_=; s/\s+/ /g; print;'

be:

echo " Hello world " | perl -ne '$_=~ s/\s+/ /g; print;'

??
| reply

tr -s

Absolutely. That's a very handy command. Thanks!
| reply
peer-to-peer

jfruh
Apple syncing patent can't come soon enough

pasmith
New Twitter features borrow from 3rd party clients

Esther Schindler
Open Source Changes the Software Acquisition Process

mikelgan
How to set up continuous podcast play on the new iTunes

David Strom
Five important Windows 7 mobility features

sjvn
Guard your Wi-Fi for your own sake                        

Sandra Henry-Stocker
Grepping on Whole Words

 

Sidekick: The Good News & the Bad News
Either way you look at it Microsoft Data Center management did not follow standards or best practices in this failure. In which case it makes me wonder more about the outsourcing of corporate data much less personal data.
- mburton325

Join the conversation here

The Daily Tip

The Daily TipQuick, practical advice for IT pros. Made fresh daily.

Hot tips:

Want to cash in on your IT savvy? Send your tip to tips@itworld.com. If we post it, we'll send you a $25 Amazon e-gift card.

Newsletters

Subscribe to ITWORLD TODAY and receive the latest IT news and analysis.

I would like to receive offers via email from ITworld partners.
By clicking submit you agree to the terms and conditions outlined in ITworld's privacy policy.
Featured Sponsor

AISO founders envisioned a Web hosting company that was environmentally friendly. While the company employed energy-efficient innovations like solar panels, its infrastructure produced unacceptable power and cooling requirements. Find out how AISO leveraged AMD technology to overcome their challenge in this case study white paper.

In this whitepaper, Scalar explores the opportunity to change the landscape with respect to mission critical databases built around Oracle. Leveraging technologies such as Linux, high-end commodity processing power and Oracle RAC technology to architect, design, build and maintain database infrastructure that delivers maximum availability, reliability and performance at a fraction of traditional cost.

On a typical day, weather.com, the Web site for The Weather Channel in Atlanta, serves up between 15 million and 20 million page views. But in September 2004, when back-to-back hurricanes ransacked Florida, the peak traffic on one day more than tripled: over 70 million page views by more than 7 million unique visitors. Read the full success story now.

Marketplace