bug#33362: System tests stuck in "shepherd[1]: waiting for udevd..."

[zimoun]

Hi Mark,

CC: Mathieu, maybe they have an insight for Cuirass.

On Fri, 26 Nov 2021 at 18:21, Mark H Weaver <mhw@netris.org> wrote:
>> On Mon, 12 Nov 2018 at 20:09, Mark H Weaver <mhw@netris.org> wrote:
>>
>>> As I write this, there are two system test builds that have been stuck
>>> for many hours, endlessly printing "shepherd[1]: waiting for udevd...":
>>>
>>>   https://hydra.gnu.org/build/3153725
>>>   https://hydra.gnu.org/build/3154365
>>>
>>> They are both on i686-linux, and on the 'core-updates' branch, but with
>>> two different build slaves (hydra.gnunet.org and guix.sjd.se).
>>>
>>> I will now abort these builds, to free up build slots for other jobs.
>>
>> I am doing bug triage and I hit this old one #33362 [1] from 2018.  It
>> is about hydra which is down now, IIRC.
>
> I doubt that this bug was about Hydra.  It guess that the bug was in our
> system test derivations.  As far as I know, the only relevance of Hydra
> to this bug is that Hydra was our CI system at that time, and therefore
> it was Hydra that brought this bug to my attention.

I agree, but I am not able to connect to the mentioned logs.  Are the
links still working?


>> Does it still make sense to keep it open?  Or can we close it?
>
> If the bug hasn't occurred recently, then I agree it's okay to close it.
>
> It would be good to check our modern Cuirass-based ci.guix.gnu.org to
> find out whether this failure mode is still occurring in our system
> tests.
>
> I see that Cuirass's web interface has improved quite significantly in
> the last couple of years, and I'm very grateful to those who've worked
> on it.  However, Cuirass still seems to be missing some important
> functionality that Hydra had.  Most notably, unless I missed something,
> it seems to lack the ability to compare the results of two evaluations
> and show the *differences* between those results, i.e. to enumerate the
> newly failing jobs, the newly succeeding jobs, and the newly aborted
> jobs.
>
> Without that functionality, it's not easy for us to notice when a job
> starts to fail, unless a user files a bug report.  The total list of job
> failures has always been too large to easily notice changes in that list
> without assistance.

I agree.  Maybe Mathieu could comment more because this missing feature
is floating around. :-)


Cheers,
simon