Re: [Savannah-hackers-public] Emacs git repository clone limits

[Bob Proulx]

Hello Philippe,

Philippe Vaucher wrote:
> I'm the author of https://hub.docker.com/r/silex/emacs, dockerized Emacs
> images for a lot of versions.
> 
> You can see one of the builds here:
> https://gitlab.com/Silex777/docker-emacs/pipelines/148568952

Wow!  I see 40 different builds there.  Which is totally awesome btw.

> The build thus builds a lot of images at once, and thus I have many
> parallel "git clone https://git.sv.gnu.org/git/emacs.git";.

That can certainly be a problem.  Because Emacs is not small.  And in
conjunction with the rest of the Internet also cloning Emacs and
cloning other projects from the server the effect can brown out the
system.

> For a while this worked fine, but since 4-5 days I get a lot of http
> 502/504 and I found out that I can only do about 4 parralel git clones.
> 
> Is there a limit and/or maintainance going? Was I but in some sort of
> throttle-list?

We do have rate limits in place.  Because otherwise the general
background radiation activity of the Internet will break things.

However nothing has changed recently in regard to the rate limits for
a long time.  As I look at the logs the last rate limit change was Sat
Dec 7 06:31:50 2019 -0500 which seems long enough ago that it isn't
anything recent.  Meaning that this is probably simply just you
competing with other users on the Internet for resources.

Cloning with the https transport that uses git-http-backend for the
backend.  We are using Nginx rate limiting.  Which you can read about
how the algorithm works here.  It is basically a smoothing process.

  https://www.nginx.com/blog/rate-limiting-nginx/

How are you doing the clone?  Is it a fresh clone into an empty
directory?  Because that would be the worst case.  In that worst case
scenario it would need to transport 100% of the repository every time.
That's pretty heavy.  The Emacs git repository is about 359MB in
size.  Seeding that with the previous version and updating it would
make that transfer much more efficient.

Whenever I have set up continuous integration builds I always set up a
local mirror of all remote repositories.  I poll with a cronjob to
refresh those repositories.  Since the update is incremental it is
pretty efficient for keeping the local mirrors up to date.  Then all
of the continuous integration builds pull from the local mirror.

This has a pretty good result in that the LAN is very robust and only
shows infrastructure failures when there is something really
catastrophic happening on the local network.  Since 100% of everything
is local in that case.

Transient WAN glitches such as routing problems or server brownouts
happen periodically and are unavoidable but then will catch up the
local mirror image with the next cronjob refresh.  Can refresh on a
pretty quick cycle.  I would do every four hours for example.  Plus
those errors are much more understandable as a network issue separate
from the CI build errors.  It's a good way to protect the CI builds
and redunce noise from them to a minimum.

Periodically the Savannah vcs server is pummeled by network abuse.
People.  It's why we can't have nice things.  I have been seeing some
zero-dark-thirty times when things have been hammered into failure.
Which then will recover by the time I wake up and go peek at them.

Do the failures are are seeing have a periodic time of day cycle when
they are more likely to happen in the middle of the US nighttime?  If
so then that is probably related.

I have been needing to look at the logs and improve the fail2ban rules
to try to push back on the abuse a little more strongly.  I do this
periodically but the type of abuse keeps changing.  For example right
now I am seeing a new wp-content path (Wordpress attack).  However
note that there is no Wordpress running on these machines.  Anti-abuse
rule maintenance is always needed.

Bob