The New parallel-tests Framework (was: Various testsuites)

[Ralf Wildenhues]

Hello once again,

allow me to expand upon this topic a bit more.  In this message, I will
not try to be fair towards the different test suite frameworks; instead,
I'll bluntly praise the new parallel-tests driver.  :-)

> Automake TESTS support, as of version 1.11, consists of two different
> test drivers that run tests from within a makefile.  The old one is
> pretty minimalistic in test suite functionality, the new one (selected
> with the `parallel-tests' option) is mostly backward-compatible, but
> also offers some more and slightly changed functionality.  It still
> operates as a set of rules within a makefile.

Here is a small tour through the features of the new framework.  I hope
you don't mind that I just post it here.



1. Getting Started
==================

First off, assume you already have tests listed in a Makefile.am file:

  TESTS = foo.test bar baz.chk ...

then just go there right now, and add the line

  AUTOMAKE_OPTIONS = 1.11 parallel-tests color-tests

in that Makefile.am file.  That's it!  Now rebuild and try it out:

  make all check

On a terminal that provides colors, the test results should be nicely
colored now, making it easy to spot failures even among several
screen-full of successful tests.


2. Inspection of Results
========================

If there are any test failures, you will notice that now, there is a
test-suite.log file which contains a summary of the tests that didn't
go as expected, including the output of the tests.

Instead of looking at the test-suite.log file for the failure output,
you could have instead also used

  make check VERBOSE=yes

to cause the output to be produced on the terminal at the end.

You will notice that there are more log files created in the build
directory: one per test, giving you an easy way to look at the output of
one particular test only.

If you happen to have rst2html installed, you can even generate a
clickable HTML log with

  make check-html


3. Running Tests in Parallel
============================

The parallel-tests option has its name from the parallel execution
feature.  If you, the developer, ensure that the tests in your suite
are independent from each other, and do not interfere while running
(for example by writing files with identical names) then you are set
to run the test suite in parallel:

  make -j4 check

Parallel runs will typically work with GNU and BSD make versions.


3. Running only a Subset of Tests
=================================

Any well-maintained test suite quickly contains lots of tests which can
take a long time to run when run in full.  So it's desirable to run only
a subset of all tests.  The parallel-tests driver provides several ways
to do this:

  a. Setting the TESTS variable:

       make check TESTS="foo.test bar"

  b. Setting the TEST_LOGS variable:

       make check TEST_LOGS="bar.log"

  c. Re-running only tests that failed during the last check:

       make recheck

  d. Re-running tests lazily:

       make check RECHECK_LOGS=

The last point (d) deserves more explanation below.  Note that with
non-GNU make, the variable overrides may require writing something
like this instead:

  env TESTS="foo.test bar" make -e check


4. Lazy test execution
======================

Say, you are right in the middle of the edit-compile-test development
cycle.  The feedback loop should be as quick as possible, so rebuilds
should update only files that are out of date, and little else.

For an example, assume `bar' is a unit test program listed in TESTS,
then you would use:

  EXTRA_PROGRAMS = bar
  CLEANFILES = $(EXTRA_PROGRAMS)

to specify that `bar' is to be compiled from sources (bar.c by default).
The old test driver would recommend using `check_PROGRAMS', but that is
not quite as lazy: `check_PROGRAMS', `check_LTLIBRARIES' and
`check_SCRIPTS' are updated before any test starts.  With EXTRA_PROGRAMS
however, we now rely on `make' dependencies to update the program just
in time for the test, possibly concurrently while other tests are
already being run.

With this in place, while editing sources for `bar' you would now run

  make check RECHECK_LOGS=

in order to recompile and rerun only those tests for which there are
newer sources in the tree.

Of course, this requires that dependencies are tracked correctly.  For
example, it is crucial that you specify in-tree library dependencies as

  bar_LDADD = ../lib/libutil.la

so that automake can infer bar_DEPENDENCIES if you don't specify that
explicitly.  Had you instead written

  # Don't do this!
  bar_LDADD = -L../lib -lutil

then a rebuild of ../lib/libutil.la would not result in a test rerun.

Note that the semantics of the `check_*' file lists have not changed:
they are still updated before any test starts.  This allows you to
specify files that themselves are not listed in TESTS, but needed
before tests can be run anyway.


5. Test Extensions
==================

The parallel-tests driver operates by creating rules for each test run.
In the above example, a rule to create `bar.log' from `bar$(EXEEXT)'
would be generated.  Now, by default, the name of the per-test log file
is the name of the test, with a possible executable extension removed,
an extension of `.test' removed, and `.log' added.  In your code, you
might have different naming conventions: Maybe you have several
extensions, or none at all.  All extensions (except for `$(EXEEXT)')
that you use should be listed in Makefile.am as

  TEST_EXTENSIONS = .test .chk

This has a few consequences:

  * The matching extension is removed from the log file name: foo.test
    creates foo.log, rather than foo.test.log.

  * With many tests, the generated Makefile will be smaller, as automake
    will use inference rules for these extensions.

  * You will be able to specify ordering relations between tests, as
    described below.


6. Dependencies between Tests
=============================

If there are dependencies between your tests, e.g., foo.test needs to be
run before baz.chk can be run, then, with TEST_EXTENSIONS set as above,
you can now go ahead and specify them as dependency relations between
the corresponding log files:

  baz.log: foo.log


7. Per-Extension Test Wrappers
==============================

Often, packages have lots of tests which are not themselves executable
programs or scripts, but some sort of data files that are to be fed to
a suitable program as input.  In order to support this more easily, the
parallel-tests driver now allows to specify wrapping programs, and flags
for these.  As a specific example, assume some tests are Perl scripts
and some are data files passed to your main program.  The scripts are to
be passed to the perl interpreter, using warnings and strict execution
by default, and your main program accepts a data file as command line
argument.  You now specify the data files as tests, and let your program
be the wrapper, the "compiler", that turns them into test logs:

  bin_PROGRAMS = main-program
  TESTS = foo.pl bar.pl circle.dat square.dat
  TEST_EXTENSIONS = .pl .dat

  PL_LOG_COMPILER = $(PERL)
  AM_PL_LOG_FLAGS = -w -Mstrict

  DAT_LOG_COMPILER = ./main-program$(EXEEXT)
  AM_DAT_LOG_FLAGS = --verbose

Tests with a listed extension `.foo' will have a "compiler" variable of
FOO_LOG_COMPILER, and two additional variables for flags, AM_FOO_LOG_FLAGS
for the developer, and FOO_LOG_FLAGS for the end-user to override.

The TESTS_ENVIRONMENT variable which was already supported by the old
test suite driver, is still supported and expanded on the command line
before `$(FOO_LOG_COMPILER)'.  However, the intention now is that the
TESTS_ENVIRONMENT variable really belongs to the end-user, to override
the environment in which tests are run.


8. Hard Errors
==============

Assume you are hacking away in your edit-compile-test cycle, and your
test suite coverage is actually ahead of your code (how it should be):
you've already implemented tests for functionality that doesn't yet work
as intended, so the corresponding tests are marked as expected to fail:

  XFAIL_TESTS = bar

Now, assume that `bar' may have different failure conditions: "normal"
failure, say, some feature that hasn't been implemented yet, and
"abnormal" failure, say, a segmentation fault that should never happen.
Such "cannot happen" situations may be caught as hard errors: if the
`bar' test exits with status 99, then it will be recorded as FAILure;
all other statuses other than zero and 77 (for SKIP) will be recorded
as eXpected FAILure, XFAIL.

This feature can be disabled by setting the variable DISABLE_HARD_ERRORS
to a nonempty value.


9. Limitations
==============

The parallel-tests driver is mostly compatible to the simple TESTS
driver.  However, it adds some inherent limitations:

  * You cannot list the same test twice and hope to have it run twice.
    This is a simple consequence of using `make' level dependencies.

  * Tests which are both generated and distributed (and updated in the
    source tree) should be created by inference rules.  You cannot use
    `$(srcdir)/' references in test names.



More details and some limitations can be found in the manual:

<http://www.gnu.org/software/automake/manual/html_node/Simple-Tests-using-parallel_002dtests.html>

Cheers,
Ralf