More spidercide

by Mark Gibbs
Be the first to comment | I like it!
More »
on this topic
April 30, 2001, 02:34 PM —  Network World — 

Last newsletter I discussed the Robot Exclusion Protocol, a method by which you can inform programs that attempt to explore your Web site (a.k.a. spiders) what you would prefer they do and don’t look at.

But where you want to exercise even finer control, you might consider using the Robots meta tag.

One of the biggest differences with the meta tag approach over REP is that meta tags don’t require a Webmaster’s involvement. You can use the Robots meta tag on any page whether or not the server has an REP robots.txt file.

Here’s an HTML page that uses the Robots meta tag:

<html>


<head>


<meta name="robots" content="noindex,nofollow">


<meta name="description" content="electric helicopters">


<title>Electric Whirlybirds</title>


</head>


<body>


A page of great information on

» posted by ITworld staff

Network World

I like it!
Post a comment
The content of this field is kept private and will not be shown publicly.
  • Allowed HTML tags: <a> <em> <strong> <cite> <code> <ul> <ol> <li> <dl> <dt> <dd>
  • Lines and paragraphs break automatically.
Free stuff

Win an Amazon Kindle!
This month's giveaway gadget - Amazon's Kindle - will keep you entertained on the long trip home to visit family and friends over the holidays. Enter the drawing now!

Applied Security Visualization
By Raffael Marty
Published by Addison-Wesley Professional
Learn more!

 

IT Manager's Handbook
By Bill Holtsnider and Brian D. Jaffe
Published by Morgan Kaufmann
Learn more!

 

Windows Vista Resource Kit
By Mitch Tulloch, Tony Northrup, and Jerry Honeycutt
Published by Microsoft Press
Learn more!

Featured Sponsor
Case Study: AISO grows its "green" business

AISO founders envisioned a Web hosting company that was environmentally friendly. While the company employed energy-efficient innovations like solar panels, its infrastructure produced unacceptable power and cooling requirements. Find out how AISO leveraged AMD technology to overcome their challenge in this case study white paper.

Download this white paper »
Myth of the Million Dollar Database

In this whitepaper, Scalar explores the opportunity to change the landscape with respect to mission critical databases built around Oracle. Leveraging technologies such as Linux, high-end commodity processing power and Oracle RAC technology to architect, design, build and maintain database infrastructure that delivers maximum availability, reliability and performance at a fraction of traditional cost.

Download this white paper »
Success Story: Weather.com handles whatever nature serves up

On a typical day, weather.com, the Web site for The Weather Channel in Atlanta, serves up between 15 million and 20 million page views. But in September 2004, when back-to-back hurricanes ransacked Florida, the peak traffic on one day more than tripled: over 70 million page views by more than 7 million unique visitors. Read the full success story now.

Download this white paper »
More Resources