Re: GC strategy on clusters

[Ludovic Courtès]

Hello!

Efraim Flashner <efraim@flashner.co.il> skribis:

> On Thu, Apr 01, 2021 at 12:17:22PM +0200, Ludovic Courtès wrote:

[...]

>>   guix package --delete-generations=4m
>> 
>> or similar, which was enough to free more space.
>
> I feel like 4-6 months should be plenty for anything active. Even if it
> were run automatically for them it wouldn't remove the last generation.

It depends.  A practical use case I have in mind: you run experiments,
you submit a paper including its results, you get initial reviews months
later, and even later it’s published and you get to present it.  At that
point, you want to answer questions and to reproduce it.  4–6 months is
not a lot in that context.

(Though of course, ideally you’d save channels.scm + manifest.scm and
share it with reviewers and readers in the first place…)

Besides, I think the whole point of Guix is that users on the cluster
can remain in control, unlike what happens with “environment modules”.

>> Longer term, I think Guix should automatically delete old generations
>> and instead store the channels + manifest to reproduce them, when
>> possible.
>> 
>
> This seems to help a bit less when we run into issues about dates being
> wrong on SSL tests, or when sources go missing.

Good points.  Hopefully “sources go missing” can soon be considered
addressed.  Really, failing TLS tests is the most worrisome issue to me
because we don’t have any idea on how to address it systematically.

> How much storage and people are you working with? Our initial multiuser
> system has 188GB for /gnu and I think 30-40 people and some people have
> profiles going back almost 3 years. Not many people have multiple
> profiles and the experiments we tried with shared profiles in
> /usr/local/guix-profiles don't see a lot of use or get updated
> frequently.

I’m not sure how much storage the Guix head node has (I’m not an admin),
but the number of users and duration is in the same ballpark.

> I guess I'm not really sure if its a technology problem or a people
> problem. Figuring out if someone is the only one pulling in a copy of
> glibc-2.25 is doable but how many copies of diffoscope is too many?
>
> On a practical note, 'guix package --list-profiles' as root currently
> lists everyone's profiles so it can be easier to see who has older
> profiles hanging around.

Actually, as non-root, I walked /var/guix/profiles/per-user on the
cluster to see the number of generations per user, which allowed us to
target those with a lot of generations.  :-)

It would be nice to provide a documented approach sysadmins could
follow!

Ludo’.