[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: Application
From: |
Technical Support |
Subject: |
Re: Application |
Date: |
Mon, 30 Jun 2003 09:08:36 -0700 (PDT) |
This is an automated email reply to acknowledge your message to
slurp@inktomi.com
Slurp is Inktomi Corporation's web-indexing robot. It collects documents
from the World Wide Web to build a searchable index for search services
using the Inktomi search engine. For more information or to see a list
of partners with whom we work, please see:
http://www.inktomi.com/products/web_search/partners.html
For some frequently asked questions on Slurp and Inktomi's web crawling
and search technology, please see later in this email.
Some web site administrators do not want robots to index their site, or
certain areas of their site. This is particularly true for sections
that contain dynamically generated pages, CGI scripts, and so on. There
is a de facto standard called the "Robots Exclusion Standard" (RES)
which allows web administrators to tell robots which areas of the site
they are allowed to visit, and which are off limits.
RES involves putting a file called "robots.txt" in the document root of
the web site, this file is parsed by a robot to determine what site
restrictions exist. Every robot that visits your site should request
the robots.txt file from your web server. If there is no robots.txt
file, the RES specifies that the robot can visit all parts of the site
if it wishes.
Slurp obeys robots.txt restrictions. Every time Slurp, visits your site it
will request the robots.txt file. If your site doesn't have a robots.txt,
Slurp will obey its own rules on which URL's to visit (for example, Slurp
doesn't visit the standard places to store CGI scripts even if robots.txt
allows it to). If you don't have a problem with Slurp's visits, or other
robots visits, then you don't need a robots.txt file. If you just want to
stop getting "robots.txt not found" errors in your server logs, you can
simply create an empty robots.txt file.
For more background on Slurp, please see:
http://www.inktomi.com/slurp.html
For more information on the RES, please see:
http://www.robotstxt.org/wc/exclusion.html
A number of frequently asked questions regarding Slurp and Inktomi's search
system are available at:
http://support.inktomi.com/Search_Engine/Product_Info/FAQ/searchfaq.html
If your question is not answered by the links above, or if you still
need further assistance, please send an email with a full problem
description to slurp-help@inktomi.com
Thank you for your interest in Slurp!
Inktomi Corporation
-------------------
- Re: Application, (continued)
- Re: Application, QWF00133, 2003/06/25
- Re: Application, nathan, 2003/06/25
- Re: Application, igor, 2003/06/25
- Re: Application, richard, 2003/06/25
- Re: Application, karl, 2003/06/25
- Re: Application, dhiebert, 2003/06/25
- Re: Application, bash-maintainers, 2003/06/25
- Re: Application, bug-gnu-utils, 2003/06/25
- RE: Re: Application, Borland Beaverton, 2003/06/25
- Re: Application, bug-gnu-gettext, 2003/06/30
- Re: Application,
Technical Support <=