Re: March update on bordeaux.guix.gnu.org

[Christopher Baines]

Ludovic Courtès <ludo@gnu.org> writes:

> Hi!
>
> Christopher Baines <mail@cbaines.net> skribis:
>
>> Related to this, I've added options to the nar-herder to help change the
>> TTL being used, and reduced the TTL for bordeaux.guix.gnu.org to 10
>> minutes (from 180 days) [4]. This will at least mean that in the future,
>> the nar-herder on bordeaux will be able to delete zstd compressed nars
>> that it's generated more quickly.
>
> It’s not 10mn right now:
>
> $ wget --debug -qO/dev/null 
> https://bordeaux.guix.gnu.org/yr39rh6wihd1wv6gzf7w4w687dwzf3vb.narinfo 2>&1 
> |grep Cache
> Cache-Control: max-age=15502374
>
>
> Or maybe that’s just for newly created nars?

The max-age of that narinfo is currently based on the scheduled removal
of the zstd compressed nar, which is going to happen quite far in the
future.

I did think of a number of ways to approach this, and I'm not sure I've
settled on the right one yet. Maybe the TTL should be capped at 600, and
then drop to 0 as the time to remove the zstd nar approaches?

This narinfo for example currently has a max-age of 600:

  https://bordeaux.guix.gnu.org/ganfjbgy75r31bwzgddpnpswwjrrffvj.narinfo

> But then again, that’s the advertised TTL; the real TTL is still
> infinite, right?

As you probably know, the situation is more complex.

The problems caused when the nar-herder started removing zstd compressed
nars shows the difference between retention of the nar in some form, and
whether a cached narinfo response can be considered fresh or stale.

Users might also not notice the availability of zstd nars if they cache
responses forever, since currently there will be a lag between the nar
becoming available, and a zstd compression being created (although we
could generate zstd compressed nars for everything).

As described below, I also do want to start removing some nars, and that
requires not having an infinite TTL.

>> I'm really unsure about the need/usefulness of narinfo caching in
>> general, the cost of storing all these narinfos locally is quite high I
>> think and I don't really know why it's done.
>
> It’s a cache.  It’s useful to have this cache because in “typical” Guix
> usage you’re likely to ask repeatedly for the same substitutes.
>
> Regarding the cost, 3f5e14182931f123c10513a3e1e2abaebfb52279 made things
> more reasonable by putting a higher bound on narinfo retention.  On my
> laptop, I have:
>
> $ ls -lrt 
> /var/guix/substitute/cache/4refhwxbjmeua2kwg2nmzhv4dg4d3dorpjefq7kiciw2pfhaf26a/
>  |wc -l
> 11549
> $ du -h 
> /var/guix/substitute/cache/4refhwxbjmeua2kwg2nmzhv4dg4d3dorpjefq7kiciw2pfhaf26a/
>  
> 50M     
> /var/guix/substitute/cache/4refhwxbjmeua2kwg2nmzhv4dg4d3dorpjefq7kiciw2pfhaf26a/
>
> Maybe that’s still excessive and we could further reduce the maximum
> caching time.

Having played around with this a bit (e.g. hacking guix weather not to
cache), I'm a bit sceptical. Given maintaining the cache takes time that
could be spent doing network I/O, and does potentially slow disk I/O, I
think it would be interesting to try and work out in what situations the
cache speeds things up overall, and in what situations it slows things
down overall..

>> 6: https://lists.gnu.org/archive/html/guix-devel/2023-05/msg00290.html
>
> BTW, should we document this mirror somewhere (and also ensure that Guix
> Foundation pays the bills), or do you view it more as an experiment for
> now?

If the project does want to provide mirrors, I think that would be
great. From this experiment, I think we have some evidence that there
are people using Guix outside of Europe, and in some cases they struggle
with the European based infrastructure. It also seems like these mirrors
do help, and the monetary cost isn't too high in my view.

I think we should probably wait until the project starts managing them
before documenting/advertising them more widely though.

>> Apart from that, the main thing on my mind for the next year regarding
>> bordeaux substitutes specifically is storage space. We're at 90%
>> capacity on hatysa (one of the two machines storing all the nars) so
>> this will need looking at shortly. I'd also like to finally get removing
>> nars that don't relate to the guix master branch happening, as that
>> should free up a little bit of space at least.
>
> Nice (the difficulty, I guess, is that some substitutes that we not
> initially for ‘master’ eventually get used on ‘master’).

Yep, my plan is to wait some long amount of time (say 6 months) before
scheduling things for removal, to check that they haven't started being
used by the master branch in this time.

We could also add other criteria as well, like tracking which nars are
generated by fixed output derivations and never removing them.

> On this issue, I think we should learn from fellow NixOS hackers.  They
> kept substitutes for ~20 years I think and are now in a difficult
> position because they cannot afford, financially, to keep that.  So one
> of the solutions envisioned was to figure out which of these millions of
> substitutes were “important” (for instance, source code), which turns
> out to be very hard if you don’t have that info already at hand.
>
>   https://discourse.nixos.org/t/nixos-s3-long-term-resolution-phase-1/36493
>
> Do you think the Data Service or another source of info would let us
> make such decisions?
>
> If we take it to the extreme, we could have a sophisticated retention
> policy like: drop all fixed-output derivations known to be available
> from disarchive.guix + SWH, drop substitutes for packages that have less
> than 100 dependents, etc.

I think the Data Service (specifically data.guix.gnu.org) might be
really helpful here, as it speeds up being able to work out what a nar
or derivation relates to.

Additionally, the nar-herder can tag narinfos (associate key=value pairs
with them), and that's intended to help you manage the nars. So we
should probably start tagging the nars with potentially useful
information now, so that we can use that data later to make desicions.

We're storing 17.5TiB of nars currently, and this increases linearly, so
it would be good to understand how this can be broken down. The
nar-herder should help here as well, as providing you can download the
11G database, that should contain all the information you need to start
digging in to this.

  wget https://bordeaux.guix.gnu.org/latest-database-dump -O bordeaux.db

Thanks,

Chris

signature.asc
Description: PGP signature