Re: On reproducing failures of MR

[Paulo Matos]

Jean Abou Samra <jean@abou-samra.fr> writes:

> Hi Paulo,
>
>
> Le 14/02/2022 à 18:50, Paulo Matos a écrit :
>> Hi,
>>
>> I am attempting to understand what are the failures in
>> https://gitlab.com/lilypond/lilypond/-/merge_requests/875
>>
>> I decided to have a baseline of test results from HEAD before applying
>> !875. For this I configured git HEAD with ./configure 
>> GUILE_FLAVOUR=guile-2.2.
>
>
>
> I guess you mean 'git master' (HEAD is the top commit of whatever branch
> you happen to be on, if you are on a branch at all).
>
>

Oh dear - of course I did that. It's not the first time and it's
embarrassing cause I work with git daily but there you go. My brain
keeps writing `git HEAD` to mean the HEAD of the default branch.

>
>> After building I ran the tests with `make CPU_COUNT=12 -j12 test`
>>
>> I get (log tail):
>> Running texi2pdf on file /tmp/tmp8dctbdf6.texi to detect default page 
>> settings.
>>
>> Dissecting...
>> Writing snippets...
>> Processing...
>> Processing 
>> /home/pmatos/dev/lilypond/build/out/lybook-testdb/snippet-names-cc1d2dc88d557a856e363b03eeaff6cc.ly
>> Command '/home/pmatos/dev/lilypond/build/out/bin/lilypond \
>>      -I /home/pmatos/dev/lilypond/input/regression/abc2ly
>> -dseparate-page-formats=ps -djob-count=12 -dseparate-log-files
>> -dinclude-eps-fonts -dgs-load-fonts --header=texidoc -I
>> /home/pmatos/dev/lilypond/Documentation/included/
>> -dcheck-internal-types -ddump-signatures -danti-alias-factor=1 -I
>> "/home/pmatos/dev/lilypond/input/regression/abc2ly" -I
>> "/home/pmatos/dev/lilypond/build/input/regression/abc2ly"
>> -daux-files -dread-file-list -dno-strip-output-dir
>> -dtall-page-formats=eps
>> "/home/pmatos/dev/lilypond/build/out/lybook-testdb/snippet-names-cc1d2dc88d557a856e363b03eeaff6cc.ly"'
>> returned non-zero exit status 1.
>> make[2]: *** [/home/pmatos/dev/lilypond/build/.././make/ly-rules.make:36: 
>> out-test/collated-files.texi] Error 1
>> make[1]: *** 
>> [/home/pmatos/dev/lilypond/build/.././make/lysdoc-targets.make:19: 
>> lysdoc-test] Error 2
>> make: *** [/home/pmatos/dev/lilypond/build/../GNUmakefile.in:316: test]
>> Error 2
>
>
>
> Phooey. Whatever the better methods to obtain what you
> want, this will need figuring out since it is important
> that new contributors don't get stuck in cryptic build
> failures.
>
>

Interestingly enough after removing the build dir and redoing it all
over, it doesn't fail. I thought it was related to configuring for guile
1.8 then configuring for guile 2.2, building and testing but I cannot
reproduce the issue now. I can actually successfully build and run the tests.

>
>> Running that command with -V does not help much:
>>
>> [...]
>> Processing `73/lily-d7b70d6a.ly'
>>
>> This returns with exit code 1.
>>
>> Has anyone seen this before?
>
>
>
> That command is not going to work in the source tree.
> You need to run it in out/lybook-testdb. Furthermore,
> it has -dseparate-log-files which is the source of the
> non-verbosity: the logging output is redirected to separate
> .log files for each .ly files. You should find information
> in out/lybook-testdb/73/lily-d7b70d6a.log.
>

That's useful to know. I certainly was not aware of that.

> Also, the build system gave you advice on how to find
> failures at the beginning of the 'make test' run. Try
> these commands.
>

Yes, I can see that now. Scrolling up is not my forte. It would
certainly be more useful to have the notice at the end such that a tail
command of the `make test` could show it, but now that I know it, it
won't bother me. :)

>
>> Or is there a better way to obtain a test
>> baseline so I can compare it with the results from !875 so that I have a
>> chance to understand if I can move the MR forward?
>
>
> Note that 'make test' does not give you a test baseline but
> just checks that all regression tests compile. To get
> a test baseline, you want 'make test-baseline'. On the other
> hand, if all you want is to see the result of a 'make check'
> run for the MR, you can just click on the middle green
> mark on the MR page (there are three of them on the right
> of 'Detached merge request pipeline #... passed') and
> scroll to the end of the log to find a link to the
> result of the make check run done automatically in the CI
> system (it will be
> https://vpetzel.gitlab.io/-/lilypond/-/jobs/1500499681/artifacts/test-results/index.html).
> As you will find, this is the same as what has been posted
> in the form of screenshots on the MR.
>

I successfully reproduced the failure in the MR.

So `make test` runs the regression tests to `out`, `make test-baseline`
creates a current baseline in `out-baseline` and `make check` runs the
tests and creates a comparison with baseline. :) I hope I got it.

> Remember, however, that existing tests passing is a necessary
> condition for a merge request to be accepted, but definitely
> not a sufficient one.
>

Makes sense. I will keep you posted on my developments in this regard.

Thanks for your time helping a new wanna-be contributor.

-- 
Paulo Matos