[Savannah-hackers-public] Re: [gnu.org #328648] lists.gnu.org and search engines

[Sylvain Beucler]

Thanks for the information.

I documented it.
https://savannah.gnu.org/maintenance/MailSystem

-- 
Sylvain

On Mon, Jan 29, 2007 at 11:29:07AM -0500, Joshua Ginsberg via RT wrote:
> If my institutional memory serves me correctly, we disallowed robots on
> lists.gnu.org because the frequency and intensity with which the spiders
> crawled lists made mail and web service on the box slow to a crawl --
> the disks were hammered and began to fry from all of the I/O. We made
> the decision that, since our archives are available and googleable from
> the numerous mirrors out there, given the choice between faster mail
> service and search-engine indexed archives, we'd prefer the former.
> 
> Lists is still under a relatively constant load. I'm attaching the load
> average graph for lists for the last 24 hours, for your viewing pleasure.
> 
> If there is not a googleable service out there that fits your needs,
> we'll look into caching mechanisms and distributed load to try to bring
> the spiders back to lists. Otherwise, this request may have to wait
> until we migrate lists to the 4-core Opteron machine.
> 
> -jag
> 
> > [beuc - Thu Jan 25 11:13:41 2007]:
> > 
> > Hi,
> > 
> > Search engines do not index lists.gnu.org (and lists.nongnu.org) at
> > all, probably due to the current robots.txt:
> > 
> >  User-agent: *
> >  Disallow: /
> > 
> > This is blocking when I try to search across several mailing lists.
> > 
> > Can you allow indexing of public archives?
> > 
> > Thanks,
> > 
> > 
> > 
> -- 
> Joshua Ginsberg <address@hidden>
> Free Software Foundation - Senior Systems Administrator
>