Removing robots from your web traffic analysis

by Sandra Henry-Stocker

May 22, 2008 —

When someone first mentioned to me that reports created by running raw access logs through software such as Analog did not meet the needs of high level management, I was caught off guard. "What could possibly be lacking?", I wondered.

If every request for a resource that a web server receives is captured in the access log, what more could anyone want? Besides, Analog annotates the most frequent visitors provides numerous useful graphs depicting such things as the most popular pages, traffic by day of the week, hour of the day, file type, file size and so on. The problem, as it turns out, was not that log files and the reports generated from them contain too little data. On the contrary, they contain too much data.

The top visitors to web sites are often robots, crawlers and spiders -- the programs that run on systems whose job it is to index the contents of the web so that the
rest of us can effectively search it. And those robots can so distort web traffic reports as to make them practically useless. If the top twenty visitors to your site are robots, reports on the top twenty visitors are not likely to tell you
whether your target audience is visiting your site. One way to produce more meaningful web traffic reports is to remove all traffic generated by robots. Fortunately,
well behaved robots will identify themselves by requesting a particular file -- the robots.txt file -- that is generally intended to instruct robots to ignore certain portions of web sites when indexing. A robots.txt, for example, might
look like this:

Disallow: /search
Disallow: /groups
Disallow: /images
Disallow: /catalogs

Properly behaved robots will request the robots.txt file and then avoid searching through any of the disallowed directories. This means that you can assume that any web client that requests robots.txt is, in fact, a robot.

Here's an example request for robots.txt:

74.6.17.155 - - [01/Apr/2008:00:03:11 +0000] "GET /robots.txt HTTP/1.0" 302 -

This access log record shows a request occuring just after midnight and a return code (302) that indicates that the requested file has not changed since it was last sent to the particular system.

A reverse lookup of the particular IP address shows us that this request came from a yahoo server, appropriately part of the crawl.yahoo.net subdomain.

# nslookup 74.6.17.155
Server:  ns1.anywhere.com
Address:  182.8.192.11

Name:    llf520181.crawl.yahoo.net
Address:  74.6.17.155

There are two problems with using requests for the robots.txt file to identify robotic traffic in your web log files. For one, the robots.txt request may not be the first request that you see from a robot in any particular log file -- especially if you rotate your log files on a frequent basis. For another, robots sometimes work together. One system might request robots.txt and share the information with other robots while they work together to index the allowable portions of the site. Removal of robot traffic can, therefore, be a little tricky.

If you're processing an access log using Perl, you can identify some of the robots by maintaining a hash of their IP addresses as they request robots.txt.

my($host,$id,$user,$date,$request,$URL,$status,$bytes)=/^(\S+) (\S+) (\S+) \
[([^]]+)\] "(\w+) (\S+).*" (\d+) (\S+)/;
# ------------------------------
# record if client is a robot
# ------------------------------
$ROBOT{$host}++ if $URL eq '/robots.txt';

As you process the remainder of your web log file, you can then ignore hits generated by robots using a statement such as this:

    # ----------------------------------------------------------------
    # skip robots
    # ----------------------------------------------------------------
    next if $ROBOT{$host};

You are likely to be able to discount the bulk of your robot-generated traffic in this manner. You might, for example, create a second access log without requests from robots, thereby elevating your human visitors to the top positions in your most frequent visitor statistics.

If you want to be more thorough, you can identify some robots by doing reverse lookups on their names. Systems with "crawl" in their names, for example, are bound to be robots. Reverse lookups, on the other hand, will dramatically slow down your processing.

You can also download a tab-delimited list of well known robot IP address ranges from here and programmatically remove their traffic.

ITworld.com