Re: running tests in parallel

[David Fang]

> > It would be nice if it were possible to run tests in parallel using
> > automake.  Waiting for tests to complete on a 4-way box while they
> > are run
> > on a single CPU is annoying.  As most 'make' implementations already
> > support running in parallel (-j), automake could just utilize this
> > functionality to runt tests in parallel.
>
> I very much agree.

Hi all,

I use a simple recipe for parallelizing tests without any modification to
automake, but it may require some semi-automatic dependence tracking to
get right.  I have the targets listed in the TESTS variable as trivial
shell scripts that are the results of other dependencies that actually run
the tests.  The gist of it:

TESTS += my_test.sh ...

# process result of test into trivial script
my_test.sh: my_test.diff
        echo "#!/bin/sh" > $@ ; \
        if test -s $< ; then \
                echo "exit 1" >> $@ ; \
        fi ; \
        chmod +x $@

# test post-processing
my_test.diff: my_test.actual my_test.expect
        diff -u $(srcdir)/my_test.expect my_test.actual > $@

# here, my_test.expect is expected to reside in the srcdir

# running the built executable on some input
my_test.actual: (input file, maybe executable too)
        (some command to make it)

The tests are only run with 'make check', but I've added some convenience
targets to run the tests without going through make check_TESTS

Since I have thousands of tests with some long dependence chains between
tests, I use many custom suffix rules and automatically compute additional
dependencies where needed.  Since most of my tests follow the
diff-and-check pattern, I end up having to append dependencies to .diff
targets, since I'm not using %-style rules.

-include diffs.depend

diffs.depend:
        cat $(list of test files through sed/awk filter) > $@

diffs.depend will look like:
test1.diff: test1.expect
test2.diff: test2.expect
test3.diff: test3.expect

while the suffix rule is:
.actual.diff:
        (deduce expect file name from $< and run diff)

This procedure sped up my tests by a perfect 2x on all of the
dual-processor machines I could get my hands on.  I tried make -j4 ...
-j32 as an experiment and watched the process tree (pstree) explode, but
didn't get much beyond 2x speedup.  I highly recommend doing this for any
reasonably large test suite.

Is this enough to work with, or should I post a more concrete example?

Fang


David Fang
Computer Systems Laboratory
Electrical & Computer Engineering
Cornell University
http://www.csl.cornell.edu/~fang/
        -- (2400 baud? Netscape 3.0?? lynx??? No problem!)