Removing robots from your web traffic analysis

When someone first mentioned to me that reports created by running raw access logs through software such as Analog did not meet the needs of high level management, I was caught off guard. "What could possibly be lacking?", I wondered.

If every request for a resource that a web server receives is captured in the access log, what more could anyone want? Besides, Analog annotates the most frequent visitors provides numerous useful graphs depicting such things as the most popular pages, traffic by day of the week, hour of the day, file type, file size and so on. The problem, as it turns out, was not that log files and the reports generated from them contain too little data. On the contrary, they contain too much data.

The top visitors to web sites are often robots, crawlers and spiders -- the programs that run on systems whose job it is to index the contents of the web so that the
rest of us can effectively search it. And those robots can so distort web traffic reports as to make them practically useless. If the top twenty visitors to your site are robots, reports on the top twenty visitors are not likely to tell you
whether your target audience is visiting your site. One way to produce more meaningful web traffic reports is to remove all traffic generated by robots. Fortunately,
well behaved robots will identify themselves by requesting a particular file -- the robots.txt file -- that is generally intended to instruct robots to ignore certain portions of web sites when indexing. A robots.txt, for example, might
look like this:

Disallow: /search
Disallow: /groups
Disallow: /images
Disallow: /catalogs

Properly behaved robots will request the robots.txt file and then avoid searching through any of the disallowed directories. This means that you can assume that any web client that requests robots.txt is, in fact, a robot.

Here's an example request for robots.txt:

74.6.17.155 - - [01/Apr/2008:00:03:11 +0000] "GET /robots.txt HTTP/1.0" 302 -

This access log record shows a request occuring just after midnight and a return code (302) that indicates that the requested file has not changed since it was last sent to the particular system.

A reverse lookup of the particular IP address shows us that this request came from a yahoo server, appropriately part of the crawl.yahoo.net subdomain.

# nslookup 74.6.17.155
Server:  ns1.anywhere.com
Address:  182.8.192.11

Name:    llf520181.crawl.yahoo.net
Address:  74.6.17.155

Sign up for ITworld's Daily newsletter
Follow ITworld on Twitter @IT_world

• Email
• Print
• Share
  • Slashdot
  • Digg
  • Stumble
  • Reddit
  • Newsvine