bug#43565: cuirass: Fibers scheduling blocked.

[Ludovic Courtès]

Hello!

Mathieu Othacehe <othacehe@gnu.org> skribis:

>> But does Cuirass create file descriptors as O_NONBLOCK?  This has to be
>> done explicitly, Fibers won’t do it for us.  As it turns out, the answer
>> is no, in at least one important case: the connection to the daemon
>> (untested patch below).
>>
>> While GC is running, Cuirass typically sends ‘build-derivations’ RPCs
>> and they block until the GC lock is released.  That can lead to the
>> situation above: a bunch of threads blocked in ‘read’ from their daemon
>> socket, waiting for the RPC reply.  OTOH, ‘build-derivations’ RPCs are
>> made from a fresh thread created by ‘build-derivations&’.
>
> While I agree not opening file descriptors with O_NONBLOCK is an issue,
> build-derivations is called in a separate thread. Blocking this separate
> thread should not block the fibers.

Agreed.

> Now the question is why there's no fetching while the GC is running? The
> answer is that "latest-repository-commit" called by "fetch-input" will
> block the only fiber dedicated to fetching. Having multiple fibers
> trying to fetch wouldn't solve anything because fetching requires some
> building from the daemon.

Exactly: when the GC lock is taken, ‘latest-repository-commit’ makes an
‘add-to-store’ RPC, and that RPC blocks.  Thus the whole fetch fiber is
blocked.

The patch should address this case.  That said, nothing useful happens
anyway when the GC lock is held, so it wouldn’t have any practical
effect.

I believe there are other cases where RPCs can be slow, for example when
there’s contention on the sqlite database.  Perhaps that could help a
bit there although again, it’s a situation where nothing useful can
happen.

> Long story short, I think we can apply your patch that can be useful to
> prevent fibers talking directly to the daemon to block, even though it
> won't help for this particular hang, that will only be fixed the GC time
> will be reduced to something more acceptable.

Yeah please go ahead if you want, or let me know if you’d rather let me
apply it.

Thanks!

Ludo’.