[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: running tests in parallel
From: |
Akim Demaille |
Subject: |
Re: running tests in parallel |
Date: |
Wed, 7 Feb 2007 21:37:20 +0100 |
It would be nice if it were possible to run tests in parallel using
automake. Waiting for tests to complete on a 4-way box while they
are run
on a single CPU is annoying. As most 'make' implementations already
support running in parallel (-j), automake could just utilize this
functionality to runt tests in parallel.
I very much agree.
I have been using for quite a while the following framework which
brings several benefits over the current framework. I wanted to
have time to write bits in Automake to avoid GNU Make syntax,
but I didn't find enough time. Maybe someone will want to pick
up the ball.
The features are:
- It can be run in parallel.
- A single run produces the quiet version on screen and
the verbose version in a log, so you no longer have to
run the tests twice. A message similar to Autotest's
invites the user to send the log to the reporting address.
- It can rely on dependencies, so if you have tests that need
not be rerun if their sources are not updated, then they are
not rerun at all. On a project of mine, this is a significant
speed up
- It is meant to produce reStructured Text, so if you produce
logs in RST too, the resulting log file is easy to convert
into clickable html.
- The display on screen uses colors to catch attention on
failures.
- It gives the path to the directory that had the failure,
which is quite handy when there are several test suites.
- The log file is not created if everything went right,
so a call to find can easily reveal the directories
where the failures occurred.
- It is meant to support several test-suites in a single
directory, but this requires help from Automake. I had in
mind something like
TEST_SUITES = test-suite-1 test-suite-2
test_suite_1_SOURCES = test1.tst test2.chk test3.tst
test_suite_1_XFAILS = test1.tst
etc. But this requires hacking Automake, and I did not take
time to do it.
Here is the file that provides these features:
check.mk
Description: Binary data
Here is an example of use that supports another extension
than .test or $(EXEEXT):
----------------------------------------------------------
# The test suite is composed of *.chk files. Each one must create a
# *.log file that we want to keep.
TESTS_ENVIRONMENT = srcdir=$(srcdir) USE_VALGRIND=$(USE_VALGRIND)
TESTS = $(notdir $(wildcard $(srcdir)/checkfiles/*.chk))
XFAIL_TESTS = \
events.chk \
lazy-test-eval.chk \
tag.chk
TEST_LOGS = $(TESTS:.chk=.log)
# .PRECIOUS: $(TEST_LOGS)
# Parallel test framework.
include $(top_srcdir)/build-aux/check.mk
# From a test file to a log file.
%.log: $(srcdir)/checkfiles/%.chk
@$(am__check_pre) $(srcdir)/uconsole-check $${dir}$< $(am__check_post)
EXTRA_DIST += uconsole-check
.PHONY: clean-test-dirs
clean-local: clean-test-dirs
clean-test-dirs:
rm -rf $(TESTS:.chk=.dir)
----------------------------------------------------------
Here is an example of textual log created this way, followed
by its HTML conversion (when running make check-html). The
idea to use RST was made by Alexandre Duret-Lutz.
test-suite.log
Description: Binary data
test-suite.html
Description: Text document
I'd be happy to help if I can do something for this to be
included in Automake.