Re: unit tests best practice

[Catonano]

Il giorno dom 6 gen 2019 alle ore 16:27 Alex Sassmannshausen <
address@hidden> ha scritto:

> Hi Cato and Mike,
>
> Thanks for the discussion — this is already shedding a ton of light :-)
>
> Catonano writes:
>
> > Il giorno gio 3 gen 2019 alle ore 19:45 Mike Gran <address@hidden> ha
> > scritto:
> >
> >> On Thu, Jan 03, 2019 at 05:53:33PM +0100, Catonano wrote:
> >> > Wat's the best practice to instrument a Guile based project for
> running
> >> > unit tests ?
> >> >
> >> > guile-git has this fragment in the Makefile.am file
> >> >
> >> > SCM_LOG_DRIVER =                                \
> >> >   $(top_builddir)/pre-inst-env                  \
> >> >   $(GUILE) --no-auto-compile -e main            \
> >> >       $(top_srcdir)/build-aux/test-driver.scm
> >> >
> >> > as you can see, what Automake produces calls a "main" procedure in a
> >> > "test-driver.scm" script in the "build-aux" folder
> >> >
> >> > The effect is that you can call
> >> >
> >> > make check
> >> >
> >> > in the root folder and have the tests run
> >> >
> >> > Instead, in the Makefile.am file produced by guile-hall wired projects
> >> you
> >> > ave this fragment
> >> >
> >> > AM_TESTS_ENVIRONMENT = abs_top_srcdir=\"$(abs_top_srcdir)\"
> >> > SCM_LOG_COMPILER = $(top_builddir)/test-env $(GUILE)
> >> > AM_SCM_LOG_FLAGS = --no-auto-compile -L \"$(top_srcdir)\"
> >> >
> >> > so there's a SCM_LOG_COMPILER instead of a SCM_LOG_DRIVER
> >> >
> >> > What is this LOG_COMPILER doing, exactly ?
> >>
> >> OK, this is a muddle, but, it goes like this.
> >>
> >>
> > Mike,
> >
> > thank you for taking the time to clarify
>
> Indeed, this is very helpful.
>
> > I think the less painful step at this stage would be to make guile-hall
> > arrange the tests in the same way that guile-git does
>
> So, currently, guile-hall does as Mike did: it uses LOG_COMPILER and
> expects the test scripts to provide an exit code.
>
> A simple but probably bad way to achieve that is to add:
> (exit (= (test-runner-fail-count (test-runner-current)) 0))
>
> to the bottom of srfi-64 test files.  The reason why it might be bad is
> that it does not generate nice exit codes, but rather simply non-zero if
> any test fails.
>
> I would be happy to have guile-hall support LOG_DRIVER instead/as well.
>
> The best way to achieve that would be to have a template driver script,
> and appropriate changes to the Makefile.am generator.
>
> We could then default to TEST_DRIVER, but provide a command line flag to
> support LOG_COMPILER.
>
> > Right now it's the other way around but I don't think Alex would
> complain,
> > he openly declares his cluelessness in the matter
>
> Quite.
>

Sorry.

No offense meant.

As for me I do feel clueless and like crawling in the mud, in addressing
these issues


> In this way, guile-hall could provide a working solution for new projects
> > without too much work in wrapping SRFFIs in order to provide compliance
> > with Automake
>
> Agreed, TEST_DRIVER would probably be closer to what people would expect.
>
> > The wrapping could me made in a further step
>
> Also agreed.
>
> Alex
>

I'm glad that you agree.

I provided an example of the "experience" that both solutions offer using a
daydream project of mine as a starting point

I present the thing here
https://gitlab.com/a-sassmannshausen/guile-hall/issues/1#note_130446033

As Mike explained, with the guile-hall provided solution, the Automake
machinery expects return codes and the Guile side doesn't provide them

So running the tests with that infrastructure does succeed but returns a
result that in my view is wrong
Unless I'm doing some mistake here.

I could try to send a patch to let guile-hall use the guil-git method in
the next few days