[Savannah-hackers] address@hidden: Your site, savannah.gnu.org, is not in our index.]

[Richard Stallman]

What is the reason for blocking bots?  Shouldn't we allow them all?

------- Start of forwarded message -------
Date: Tue, 20 Nov 2001 19:15:11 -0800
From: "GoogleBot" <address@hidden>
To: address@hidden
Cc: address@hidden, address@hidden
Subject: Your site, savannah.gnu.org, is not in our index.


Dear Webmaster,

Your site, savannah.gnu.org, is currently blocking access 
to search engine crawlers with a robots.txt file 
( http://savannah.gnu.org/robots.txt ). As a result, we have not 
been able to add your site to our index and cannot point our users to your
pages. If that is your intention, feel free to ignore the rest of this
email. If however, you wish to have us send users to you, please add
these two lines to your robots.txt file:

User-agent:GoogleBot
Disallow:

If you'd like us to crawl your entire site, the "Disallow:" command
should be on a line by itself, with nothing following the ":".

If there are areas of your web site you would rather we didn't crawl,
you can tell us that by adding specific Disallow commands. For example,
to prevent us from crawling any pages under http://savannah.gnu.org/cgi-bin/ 
include the following lines in your robots.txt file:

User-Agent:GoogleBot
Disallow: /cgi-bin/

For more details on robots you can check out
http://www.robotstxt.org/wc/robots.html

We hope you found this information helpful and look forward to sending
our inquisitive Google users your way.

The Google Team
------- End of forwarded message -------