Re: unit tests best practice

[Mike Gran]

On Thu, Jan 03, 2019 at 05:53:33PM +0100, Catonano wrote:
> Wat's the best practice to instrument a Guile based project for running
> unit tests ?
> 
> guile-git has this fragment in the Makefile.am file
> 
> SCM_LOG_DRIVER =                                \
>   $(top_builddir)/pre-inst-env                  \
>   $(GUILE) --no-auto-compile -e main            \
>       $(top_srcdir)/build-aux/test-driver.scm
> 
> as you can see, what Automake produces calls a "main" procedure in a
> "test-driver.scm" script in the "build-aux" folder
> 
> The effect is that you can call
> 
> make check
> 
> in the root folder and have the tests run
> 
> Instead, in the Makefile.am file produced by guile-hall wired projects you
> ave this fragment
> 
> AM_TESTS_ENVIRONMENT = abs_top_srcdir=\"$(abs_top_srcdir)\"
> SCM_LOG_COMPILER = $(top_builddir)/test-env $(GUILE)
> AM_SCM_LOG_FLAGS = --no-auto-compile -L \"$(top_srcdir)\"
> 
> so there's a SCM_LOG_COMPILER instead of a SCM_LOG_DRIVER
> 
> What is this LOG_COMPILER doing, exactly ?

OK, this is a muddle, but, it goes like this.

LOG_DRIVER and LOG_COMPILER come from Automake.  If you use them,
Automake will generate "make check" rules for all the test scripts in
TESTS.

When you use LOG_COMPILER, Automake considers a test to pass if it
returns an exit status of 0, perhaps by calling '(exit 0)' in Guile.
Exit code 1 = failure. 77 = skip.  99 = fatal eror.  So your TESTS
scripts as run by the command line in LOG_COMPILER need to return
proper exit codes.  Automake will also automatically gather anything
your tests print into .log files and .trs (test result files).

If you use LOG_DRIVER, Automake expects your scripts to manually
create its .log and .trs files.  You use LOG_DRIVER if the simple
exit-code-based pass/fail criteria isn't good enough.  Look
in the Automake docs under 'Custom Test Drivers' for info.

> 
> This is what the test-env file contains
> 
> #!/bin/sh
> 
> \"@abs_top_builddir@/pre-inst-env\" \"address@hidden"
> 
> exit $?
> 
> what is this ?
> 
> What does this do ?

Typically these pre-inst-env scripts set up the GUILE_LOAD_PATH for
when you are running the tests in the development tree before
installing them.  (There is no standard pre-inst-env script; there are
a few different versions.)  GUILE_LOAD_PATH works better with absolute
paths.  It may also set compiled paths, environment variables, or
LD_LIBRARY_PATH.

> 
> Alex admits he goes full cargo culting when dealing with the Autotools, in
> fact there's this comment line at the beginning of the section producing
> the Autotools related files in guile-hall
> 
> ;;;; Full on cargo cult!
> 
> So, if there's any kind soul here willing to clarify/illustrate what is
> going on we could make the effort to condensate such knowledge in
> guile-hall
> 
> Improving the experience in dealing with Guile based project would be a
> great progress
> 
> One last thing
> 
> test-driver uses the API provided by srfi-64 to actually run the tests and
> display the results
> 
> Does Guile provide a standard version of such script ?
> Should Guile provide one ?

I should note that srfi-64 doesn't work out of the box with
LOG_COMPILER because it doesn't return the proper exit status on
failure, either 0 = pass, 1 = fail, 77 on skip, 99 on hard error.  It
isn't that complicated to add that functionality in, and that would be
a great project for someone to do.

For the one project were I tried to make a full test suite
(guile-ncurses), I went with lots of tiny scripts and used the
LOG_COMPILER exit code strategy.  For that I make a small test library
here that converts #t and #f from a procedure into exit codes.

http://git.savannah.gnu.org/cgit/guile-ncurses.git/tree/test/automake-test-lib.scm

Regards,

Mike Gran