Re: [RFC] Automatic 'make check' in CI

[Jonas Hahnfeld]

Am Freitag, den 20.11.2020, 10:11 +0100 schrieb Han-Wen Nienhuys:
> > There are a few known issues that I'm aware of:
> >  - I needed to delete input/regression/option-help.ly because it
> > logs the options currently in use by lilypond, including the job-
> > count which varies when using our own runners with more than one
> > core.
> > 
> 
> we could overwrite the job-count option in the option help, as it is
> irrelevant by the time you get to the file processing. 

While this might work for job-count (I'd still consider it a hack),
there is at least also log-file which depends on the current process.
The right solution would be to show the original default value in the
help output, but the test prints exactly what you enter into
scm/lily.scm so I don't really see its benefit at this point..

>  
> >  - Sometimes the test-results contain spurious diffs, for example
> > here:
> > https://hahnjo.gitlab.io/-/lilypond/-
> > /jobs/858441670/artifacts/test-results/index.html
> > I can reproduce this locally with --enable-checking, but haven't
> > investigated further yet (there were a couple of other problems
> > that I needed to solve in order to get things working...). If
> > somebody has an idea for a fix, that would be great but I think
> > these can be safely ignored for now.
> > 
> 
> This could happen because of false-positives in the conservative
> garbage collection. It's not super-likely, but at the same time, it
> can't be ruled out. Does it always happen with the same files?  I
> have never seen this, and the files are not doing anything out of the
> ordinary.

It doesn't happen every time and AFAICT it might hit many different
files, but I see it pretty consistently when building with --enable-
checking.

> IIRC, the dead-object detection can't be made to work anyway with
> GUILE 2.x, so this might be a good moment to scrap it.

As I said, I didn't investigate yet. But an explicit warning that
consistently triggers on different systems at least makes me wonder if
there's something going on.
 
> > Despite these shortcomings, I think it would make sense to enable
> > this first implementation in lilypond/lilypond. What do you think?
> > 
> 
> yes, +1 . I am especially happy that this will likely have no extra
> overhead.

No extra overhead per pipeline in MRs, but you get an extra pipeline
for every merge to master. We'll see if this poses a serious problem...

Jonas

signature.asc
Description: This is a digitally signed message part