Re: robots.txt in git and online are not the same

[Mark Polesky]

If robots.txt was getting updated properly, all of our Google
search bar problems would be solved.  We could then stop telling
Google to restrict the search results to a patrticular version
from the search box itself.  The robots.txt file only allows the 
current stable docs to be indexed.  

If robots.txt was getting updated properly, then only 2.16 should
appear in search results, and would continue to appear for a
little while after we release 2.18.  Then, the next time the 
Google bot visits lilypond.org, it would read robots.txt, remove
the 2.16 docs from the search results, then follow every link from
the homepage that isn't "disallowed", i.e. the entire website,
with only the 2.18 docs.  That's exactly what we want, so let's
just fix the problem with robots.txt.

By the way, fixing that would kill 3 items in the tracker with one 
blow:

Issue 2909: Manual search returns results from wrong version
http://code.google.com/p/lilypond/issues/detail?id=2909

Issue 3209: Searching stable release documentation should only return results 
from stable release
http://code.google.com/p/lilypond/issues/detail?id=3209

Issue 3367:   Web/Docs: LilyPond version is not clear on docs web pages
http://code.google.com/p/lilypond/issues/detail?id=3367

- Mark




----- Original Message -----
> From: Mark Polesky <address@hidden>
> To: bug-lilypond <address@hidden>
> Cc: 
> Sent: Friday, June 28, 2013 11:37 AM
> Subject: robots.txt in git and online are not the same
> 
> For some reason, http://www.lilypond.org/robots.txt differs
> from Documentation/web/server/robots.txt as found in git.  
> It looks as if it isn't being updated on the server?  I 
> don't get it, but it's worth looking into, since this helps
> remove outdated links when someone googles us from the
> outside.
> 
> - Mark
> 
> 
> _______________________________________________
> bug-lilypond mailing list
> address@hidden
> https://lists.gnu.org/mailman/listinfo/bug-lilypond
>