Re: regtest not catched by make check

[Thomas Morley]

Am Sa., 7. Nov. 2020 um 19:29 Uhr schrieb Jonas Hahnfeld <hahnjo@hahnjo.de>:
>
> Am Samstag, den 07.11.2020, 17:30 +0100 schrieb Thomas Morley:
> > Am Sa., 7. Nov. 2020 um 15:35 Uhr schrieb Michael Käppler 
> > <xmichael-k@web.de>:
> > >
> > > Am 07.11.2020 um 13:23 schrieb Thomas Morley:
> > > > Hi,
> > > >
> > > > it would be nice to have a regression test for
> > > > https://gitlab.com/lilypond/lilypond/-/merge_requests/497
> > > >
> > > > I think I found some way to display differently with and without the
> > > > fix. See the attached file.
> > > >
> > > > Though, I can't convince 'make check' to catch the difference.
> > > Hi Harm,
> > > could you please try to run
> > > `lilypond -ddump-signatures' on your regtest, with and without the fix
> > > and compare
> > > the .signature files?
> > >
> > > Cheers,
> > > Michael
> > > >
> > > > Currently I'm at a loss. Any hints?
> > > >
> > > > Thanks,
> > > >    Harm
> > >
> >
> > Hi Michael,
> >
> > I followed your advice and applied a diff between the two .signature-files.
> > With no output. They are equal...
> >
> > Though, the pdf's aren't. See attached.
> >
> > Obviously I don't understand what 'make chack' is looking for...
>
> As far as I understand `write-system-signature' in scm/stencil.scm, the
> signature files have the positions of all grobs and their extents
> *before* applying stencils.
> What about just having a simple test that would fail visual inspection?
> When looking through the list of all regression tests, it should be
> pretty catchable if the description says that all objects have a
> certain color, but they don't.
>
> Jonas

To be honest, before, I had not thought very deeply about how
comparing regression tests with 'make  check' works.

But now I think a regression test should actually _test_ something.
The test may succeed or fail.
If it fails, (but the snippet itself compiles), then the result should
be catched by 'make check'. In cases where this can't be made
possible, the snippet should error, stop 'make check' and a meaningful
error message should be returned.

In the current case I finally found something catchable, basically I
print a Staff with separated markups (_not_ a single markup). The
entire markups are sized differently if the test succeeds or fails.

I'll upload a new patch tomorrow, very late here.

Thanks,
  Harm