Re: Cannot clone from anonymous git

[Bob Proulx]

Eduardo A. Bustamante López wrote:
> When I try to clone:
> |  dualbus@hp ~ % git clone git://git.savannah.gnu.org/bash.git
> |  Cloning into 'bash'...
> |  fatal: Could not read from remote repository.

Connection was closed immediately.

> |  Please make sure you have the correct access rights
> |  and the repository exists.

Misleading error message.  It couldn't connect.

> The links here appear to work: http://git.savannah.gnu.org/cgit/bash.git

Since Friday the 6th there was a large move with various assorted
related and unrelated changes.  Which has makes it difficult to tell
what broke what.  I am deep in the middle of *.savannah and can give
the most information about it.  Immediately after the move there was a
systematic breakage of uids and ACLs which were fixed last weekend.

But the problem is only partially Savannah.  Most of the problem is
global network connectivity.  It seems to be worse from Europe than
from the US.  For example me in Colorado I rarely have any problem.
But at the same time people overseas report complete failures.  There
are reports of problems across all of the servers from fencepost to
lists.gnu.org to the web server systems.  The common change to all of
them was a core router change.

On Savannah vcs the network connectivity tickles a problem with the
git daemon.  It causes them to get stuck.  This causes them to hang
around endlessly.  They are launched from xinetd with 100 process
limit.  Especially during busy activity they fill up the slots.  Then
xinetd has a bug IMNHO that instead off allowing connections to queue
it simply closes any additional connection.  This is what you see and
describe in the above.  git:// failed because all 100 slots were in
use and xinetd closed your additional connection.  Meanwhile http://
worked because that used apache.  Maybe the problem is actually the
xinetd which is common to the cvs, hg, bzr, svn daemons which are also
suffering.

Now before everyone decided to switch to apache note that apache also
has process limits.  It just so happens that fewer people are using it
and so it wasn't limited.  Yet.  If everyone switched to apache
http:// then it would simply switch the problem from one daemon to the
other.  We see all of the source daemons stack up.  This includes cvs,
hg, bzr, and svn processes too.  Something network related is causing
them to hang around after they should have exited.  I have been
tweaking the tcp keepalives to try to makedead the stale processes
faster.  Assaf has been doing the majority of the process triage and
manually pruning processes in order to keep things functional.

On the client side it is impossible to distinguish between git daemons
consuming the limit of process slots versus router connectivity
problems.  Both have the same effect.  However if you can get through
with ssh:// or http:// then that indicates xinetd process slots are
full.  But over the last week (mostly Europe and other overseas
clients) reports endless problems trying to post bugs using the web
server.  Endless problems trying to SMTP mail in and out.  Reports of
rsync failing.  Sometimes IPv4 works when IPv6 is failing.  All of
those are separate except for the global network connectivity through
the routers.

Obviously the most urgent problem to fix is the global network
connectivity problem.  The FSF admins are aware of the problem.  They
post (terse) status updates to https://pumprock.net/fsfstatus so do
please keep an eye on it for out-of-band status updates.  Especially
when there are Boston local network problems.

If you are suffering problems I encourage a trouble report being made.
Please include details.  Say where from and where to.  Say whether it
was IPv4 or IPv6.  Time the problem occurred.  Then if later it works
update and say that.  Because one of the problems is that this seems
worse from Europe than from the US.  There isn't a reproducible test
case to show if a change fixes things or not.  I in Colorado have a
hard time triggering any problems but have been able to on occasion.
But people in Europe have been most of the problem reports.  Most of
them have been using IPv6.

And that is all I know.

Bob