March update on bordeaux.guix.gnu.org

[Christopher Baines]

Hey!

The last update was sent out back in January [1], while things are
stable there has been some activity in the last couple of months.

1: https://lists.gnu.org/archive/html/guix-devel/2023-01/msg00199.html


## Numbers

bordeaux.guix.gnu.org currently provides ~2.2 million nars, which take
up ~9.2TiB to store.

Substitute availability is still reasonable, although recent
availability for i686-linux has been lower than it has been previously.


## Mirrors

Still no progress has happened in terms of mirrors. This would still be
nice to get setup, but it's probably not the priority in terms of
administration and infrastructure.


## Serving fixed output files by hash

bordeaux.guix.gnu.org is now in the list of content addressed mirrors
[2].

2: 
https://git.savannah.gnu.org/cgit/guix.git/commit/?id=7d0ebe040d80adcf143656e754a82b569243568c

In addition to the above change, some fixed output derivations use
download-nar rather than content addressed mirrors, so I've now updated
that to also fetch from bordeaux.guix.gnu.org [3].

3: 
https://git.savannah.gnu.org/cgit/guix.git/commit/?id=b59f89cf18fbad9ee95521c4cadc6642c580feb8


## Referential integrity

This is usually something I think about in the context of databases, but
nars/narinfos have references, and I've been working to ensure that for
any nar provided by bordeaux.guix.gnu.org, you're also able to fetch any
nar it references.

I noticed something was lacking here a while ago when I went to
substitute an output for guix pull, but guix wasn't able to since some
referenced outputs weren't available. These missing outputs turned out
to be files that guix inserts in to the store to use as sources in the
derivations computed by guix pull. The build coordinator now knows about
this and will publish these as nars by default [4]. This was never an
issue when actually using guix pull, since it would compute the
derivations and create the missing outputs locally, so they aren't
usually substituted. To backfill the missing items, I wrote a script to
fetch the nars from data.guix.gnu.org, since it has them as part of
providing substitutes for the derivations.

4: 
https://git.savannah.gnu.org/cgit/guix/build-coordinator.git/commit/?id=751910162c54d0bf85fa5a21c25ad229cb12828d

There's still a bit more work to do in terms of ensuring outputs aren't
missing when nars are inserted and removed, but this is nearly there.


## Offering zstd compressed nars

bordeaux.guix.gnu.org uses lzip to compress the nars, which is good
choice in terms of minimising the storage requirements and size of
downloads. It's been known for a while though that this might not suit
all users, as those with fast network connections may benefit from nars
that can be decompressed much quicker even if there's more bytes to
download.

5: https://guix.gnu.org/en/blog/2021/getting-bytes-to-disk-more-quickly/

To enable providing zstd compressed nars, without having to store every
nar as an lzip compressed file and zstd compressed file, the nar-herder
now supports generating cached nars with different compressions.

The nar-herder running on bayfront (which serves bordeaux.guix.gnu.org)
is now configured to generate and cache zstd nars [6]. Currently there
are ~25,000 cached nars taking up about 86GiB of space.

6: 
https://git.savannah.gnu.org/cgit/guix/maintenance.git/commit/?id=c4778aa2689e0e73e77cadb71a04dc2dfee4ea27


## Next steps

Last time I said that I thought the main blocking issue in making
bordeaux.guix.gnu.org the default substitute server to try first was the
lack of zstd nars. Some nars are now available with zstd compression, so
there's progress been made on this at least. I think a good argument can
be made for trying bordeaux.guix.gnu.org being better for users than
trying ci.guix.gnu.org first, but it's not clear cut.

On the issue of expanding the storage for nars, I've submitted the
request to purchase an additional hard drive for hatysa to expand the
storage there. There is still a need to come up with a plan to replace
bishan when it runs out of space (ideally before!).

Now that bordeaux.guix.gnu.org is building things for the master branch,
plus qa.guix.gnu.org is submitting lots of builds for patches and
branches, there's more going on at varying priorities. This is really
good, but there's a greater need now to expose what's being built and
what the backlog looks like.

If you're interested in working on any of this, do let me know as while
I don't have time to work on much of it myself, I should be able to make
time to help others.

Let me know if you have any comments or questions!

Thanks,

Chris

signature.asc
Description: PGP signature