More spidercide

by Mark Gibbs

April 30, 2001 —

Last newsletter I discussed the Robot Exclusion Protocol, a method by which you can inform programs that attempt to explore your Web site (a.k.a. spiders) what you would prefer they do and don’t look at.

But where you want to exercise even finer control, you might consider using the Robots meta tag.

One of the biggest differences with the meta tag approach over REP is that meta tags don’t require a Webmaster’s involvement. You can use the Robots meta tag on any page whether or not the server has an REP robots.txt file.

Here’s an HTML page that uses the Robots meta tag:

<html>

<head>

<title>Electric Whirlybirds</title>

</head>

<body>

A page of great information on

Network World