Re: parallel testsuite execution

[Akim Demaille]

>>> "RW" == Ralf Wildenhues <address@hidden> writes:

 > Hello Akim,

Hi Ralf,

 >> So I would like to be able to write something like
 >> 
 >> TEST_SUITES = foo bar
 >> foo_TESTS = foo1.chk foo2.test foo3
 >> bar_TESTS = bar1.test bar2.sh bar3.c
 >> 
 >> and be able to write on the side rules explaining how to .chk -> .log,
 >> .test -> .log etc.

 > Sounds useful.

 >> Maybe instead of _TESTS we could reuse _SOURCES?  That's not
 >> inconsistent: they are really sources which must be compiled into
 >> *.log files which are then linked together to produce foo.log and
 >> bar.log.

 > Hmm.  That would prevent us to name test suites after programs.
 > As I understand your proposal above, that would be intended though.

Actually the test suite don't really exist, what does exist is the set
of programs to run, their associated logs, and the resulting log of
the test suite.  So actually I could have written:

     TEST_SUITES = foo.log bar.log

and maybe

     foo_log_TESTS = foo1.chk foo2.test foo3
     bar_log_TESTS = bar1.test bar2.sh bar3.c



 > I don't remember whether you turned them off for non-terminal output,
 > but it should be configurable, and default to on only for terminal
 > output.

That's what I do, indeed:

# If stdout is a tty, use colors.  If test -t is not supported, then
# this fails; a conservative approach.  Of course do not redirect
# stdout here, just stderr...
am__tty_colors =                                \
if test -t 1 2>/dev/null; then                  \
  red='ESC[0;31m';                              \
  grn='ESC[0;32m';                              \
  lgn='ESC[1;32m';                              \
  blu='ESC[1;34m';                              \
  std='ESC[m';                                  \
fi


Here is the latest version we use.  Benoît Sigoure has added another
feature: so called "hard errors", errors that cannot be saved by an
XFAIL.  This can be used for instance when you do not tolerate a SEGV,
even if the test is Failed.

check.mk
Description: Binary data

 >> Sure, but who's going to spend some time on this?

 > Which of the two?  The Sun make adjustments are typically not hard.
 > The ?= is really GNU make specific.

These are quite trivial.  The main problem is supporting

     TEST_SUITES = foo.log bar.log
     foo_log_TESTS = foo1.chk foo2.test foo3
     bar_log_TESTS = bar1.test bar2.sh bar3.c