[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: Building and caching old Guix derivations for a faster time machine
From: |
Ludovic Courtès |
Subject: |
Re: Building and caching old Guix derivations for a faster time machine |
Date: |
Wed, 22 Nov 2023 19:27:18 +0100 |
User-agent: |
Gnus/5.13 (Gnus v5.13) |
Hi,
Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:
>> I agree. The ‘guix publish’ TTL¹ at ci.guix was increased to 180 days
>> following <https://issues.guix.gnu.org/48926> in 2021. That’s still not
>> that much and these days and right now we have 84 TiB free at ci.guix.
>>
>> I guess we can afford increasing the TTL, probably starting with, say,
>> 300 days, and monitoring disk usage.
>>
>> WDYT?
>
> While the 84 TiB we have at our disposal is indeed lot, I'd rather we
> keep the TTL at 180 days, to keep things more manageable for backup/sync
> purposes. Our current TTL currently yields 7 TiB of compressed NARs,
> which fits nicely into the hydra-guix-129 10 TiB slice available for
> local/simple redundancy (it's still on my TODO, missing the copy bit).
>
> I've been meaning to document an easy mirroring setup for that
> /var/cache/guix/publish directory, and having 14 TiB instead of 7 TiB
> there would hurt such setups.
Maybe we should learn from what Chris has been doing with the
Nar-Herder, too. Ideally, the build farm front-end (‘berlin’ in this
case) would be merely a cache for recently-built artifacts, and we’d
have long-term storage elsewhere where we could keep nars for several
years.
The important thing being: we need to decouple the build farm from
(long-term) nar provision.
> Perhaps a compromise we could do is drop yet another compression format?
> We carry both Zstd and LZMA for Berlin, which I see little value in; if
> we carried only ZSTD archives we could probably continue having < 10 TiB
> of NARs for a TTL of 360 days (although having only 3.5 TiB of NARs to
> sync around for mirrors would be great too!).
>
> What do you think?
For compatibility reasons¹ and performance reasons², I would refrain
from removing lzip or zstd substitutes, at least for “current”
substitutes.
For long-term storage though, we could choose to keep lzip only (because
it compresses better). Not something we can really do with the current
‘guix publish’ setup though.
Thoughts?
Ludo’.
¹ Zstd support was added relatively recently. Older daemons may support
lzip but not zstd.
² https://guix.gnu.org/en/blog/2021/getting-bytes-to-disk-more-quickly/