bug#49439: grafts cause “guix environment” to get killed with OOM

[Sarah Morgensen]

Hi,

Ludovic Courtès <ludo@gnu.org> writes:

> Hi,
>
> Ludovic Courtès <ludo@gnu.org> skribis:
>
>> Thinking about it, the grafts code depends on what’s in the store: when
>> nothing is in the store, it bounces to the “build handler”, which
>> accumulates the list of missing store items, until it starts building
>> them.
>
> So I can reproduce the problem Ricardo and you initially reported by
> running:
>
>   ./pre-inst-env guix environment pigx-scrnaseq --search-paths
>
> after removing some of the ungrafted store items with:
>
>   guix gc -D $(guix build r-rlang --no-grafts) \
>     $(guix gc --references $(guix build pigx-scrnaseq --no-grafts))

Same here. I'm glad we were able to pinpoint this!

>
> The seemingly endless CPU usage and unbound memory use comes from the
> ‘build-accumulator’ build handler.  Here, the graph of ‘pigx-scrnaseq’
> has many nodes, and many of them depend on, say, ‘r-rlang’.  Thus, when
> ‘r-rlang’ is not in the store, the grafting code keeps asking for it by
> calling ‘build-derivations’, which aborts to the build handler; the
> build handler saves the .drv file name and the continuation and keeps
> going.  But since the next package also depends on ‘r-langr’, we abort
> again to the build handler, and so on.
>
> The end result is a very long list of <unresolved> nodes, probably of
> this order in this case:
>
>   $ guix graph -t reverse-package r-rlang |grep 'label = "'|wc -l
>   594
>
> Presumably, the captured continuations occupy quite a bit of memory,
> hence the quick growth.
>
> I suppose one solution is to fire suspended builds when the build
> handler sees a build request for a given derivation for the second time.
> It needs more thought and testing…

I wonder if instead it's possible to reduce the memory taken by the
continuations? As someone who has absolutely no experience with the
build/derivation system, it seems like all we *should* need to save is
the derivation file name.

>
> Ludo’.
>
> PS: Did you know ‘pigx-scrnaseq’ has twice as many nodes as
>     ‘libreoffice’?
>
>       $ guix graph -t bag pigx-scrnaseq |grep 'label = "'|wc -l
>       1359
>       $ guix graph -t bag libreoffice |grep 'label = "'|wc -l
>       699
>
>     That makes it a great example to study and fix scalability issues!

For those interested, I've compiled a list of the top 60
highest-dependency packages, as reported by package-closure:

pigx                    1630
nextcloud-client        1539
akregator               1486
kmail                   1484
korganizer              1481
kdepim-runtime          1480
kincidenceeditor        1478
keventviews             1477
kmailcommon             1476
kcalendarsupport        1475
kmessagelib             1474
knotes                  1472
kaddressbook            1469
libksieve               1467
kdepim-apps-libs        1465
libgravatar             1463
kpimcommon              1462
akonadi-calendar        1453
pigx-bsseq              1448
elisa                   1446
kaffeine                1432
kdenlive                1431
kmailtransport          1431
dolphin-plugins         1426
k3b                     1424
libkgapi                1422
dolphin                 1421
kopete                  1403
pigx-sars-cov2-ww       1401
krdc                    1398
baloo-widgets           1397
baloo                   1396
pigx-chipseq            1396
krfb                    1389
ffmpegthumbs            1388
kget                    1382
kmplayer                1381
kdelibs4support         1375
pigx-scrnaseq           1342
kdevelop                1340
kmailimporter           1326
libkdepim               1325
pigx-rnaseq             1324
labplot                 1316
smb4k                   1315
kleopatra               1311
kalarmcal               1311
choqok                  1311
okular                  1310
ktnef                   1310
ktorrent                1310
kate                    1308
akonadi-search          1308
kcalutils               1307
yakuake                 1306
khelpcenter             1305
libksysguard            1305
kdeconnect              1304
konsole                 1304
libkleo                 1304

There seem to be a lot of kde packages in there, so perhaps this issue
isn't as rare as we might otherwise expect?

--
Sarah