Re: regtest not catched by make check

[Jonas Hahnfeld]

Am Sonntag, den 08.11.2020, 01:49 +0100 schrieb Thomas Morley:
> Am Sa., 7. Nov. 2020 um 19:29 Uhr schrieb Jonas Hahnfeld <hahnjo@hahnjo.de>:
> > 
> > Am Samstag, den 07.11.2020, 17:30 +0100 schrieb Thomas Morley:
> > > Am Sa., 7. Nov. 2020 um 15:35 Uhr schrieb Michael Käppler 
> > > <xmichael-k@web.de>:
> > > > 
> > > > Am 07.11.2020 um 13:23 schrieb Thomas Morley:
> > > > > Hi,
> > > > > 
> > > > > it would be nice to have a regression test for
> > > > > https://gitlab.com/lilypond/lilypond/-/merge_requests/497
> > > > > 
> > > > > I think I found some way to display differently with and without the
> > > > > fix. See the attached file.
> > > > > 
> > > > > Though, I can't convince 'make check' to catch the difference.
> > > > Hi Harm,
> > > > could you please try to run
> > > > `lilypond -ddump-signatures' on your regtest, with and without the fix
> > > > and compare
> > > > the .signature files?
> > > > 
> > > > Cheers,
> > > > Michael
> > > > > 
> > > > > Currently I'm at a loss. Any hints?
> > > > > 
> > > > > Thanks,
> > > > >    Harm
> > > > 
> > > 
> > > Hi Michael,
> > > 
> > > I followed your advice and applied a diff between the two 
> > > .signature-files.
> > > With no output. They are equal...
> > > 
> > > Though, the pdf's aren't. See attached.
> > > 
> > > Obviously I don't understand what 'make chack' is looking for...
> > 
> > As far as I understand `write-system-signature' in scm/stencil.scm, the
> > signature files have the positions of all grobs and their extents
> > *before* applying stencils.
> > What about just having a simple test that would fail visual inspection?
> > When looking through the list of all regression tests, it should be
> > pretty catchable if the description says that all objects have a
> > certain color, but they don't.
> > 
> > Jonas
> 
> To be honest, before, I had not thought very deeply about how
> comparing regression tests with 'make  check' works.
> 
> But now I think a regression test should actually _test_ something.
> The test may succeed or fail.
> If it fails, (but the snippet itself compiles), then the result should
> be catched by 'make check'. In cases where this can't be made
> possible, the snippet should error, stop 'make check' and a meaningful
> error message should be returned.

While this would be ideal, it might not always be possible. That's
exactly why visual inspection may be required.

> In the current case I finally found something catchable, basically I
> print a Staff with separated markups (_not_ a single markup). The
> entire markups are sized differently if the test succeeds or fails.

This sounds like the regtest is getting even more complex than the
initial code you shared in this thread. You should weight if this is
worth the benefit it provides; test code should usually so trivial that
it doesn't need testing itself.
FWIW fiddling around with custom stencils just to get something 'make
check' can recognize is the wrong trade-off IMO.

Jonas

> I'll upload a new patch tomorrow, very late here.
> 
> Thanks,
>   Harm

signature.asc
Description: This is a digitally signed message part