[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Extended test suites
Extended test suites
Thu, 27 Aug 2009 18:33:05 +0200
I have a project based on automake which comes with about 120 tests. The
bad thing is that some of the tests sometimes fail, but not always. I
would like to be able to express the fact that some tests are absolutely
mandatory to release the product while others are not. I could then say
that my product is entirely stable on linux/x86, but that some less
common features are not stable on mac osX for example. The remaining
bugs on the less common features would not be release-blockers.
I already know about the XFAIL variable and the possibility to mark that
some tests are known to fail (and unfortunately, we also have to use
this). But I'm here speaking of something like a TEST_EXTRA variable and
a check-extra: make target testing the base tests plus the extra ones.
I tried to do it with something like the following Makefile.am chunk:
TESTS?=regular mandatory tests
TESTS_EXTRA=other, more fragile tests.
for d in $(SUBDIRS) ; do $(MAKE) check-extra $$d ; done
TESTS="$(TESTS) $(TESTS_EXTRA)" $(MAKE) check
The benefit of this idea would be to reuse automatically the good points
of the regular check target on my extended set of tests. But it does not
work because automake seem to not detect that I set the TESTS variable
if I use the ?= construct instead of =. The check variable is then
absent from the generated Makefile.in. But I need this construct to be
able to override the list of tests.
Another approach would be to dupplicate the check target in my code to
deal with the content of $(TESTS_EXTRA), but it seems rather inelegant
So, here is my questions:
* Do you guys think that including a notion of TESTS_EXTRA would be
interesting for other automake users (I'm not quite sure of this myself)
* Could you please change the test detection mecanism so that I could
override the test list as I wanted to do?
* Have other people faced this issue and used a more elegant solution
than this would TESTS_EXTRA story?
Thanks for your time, and for this incredibly useful tool.
A language that doesn't affect the way you think about programming, is
not worth knowing. -- "Epigrams in Programming", by Alan J. Perlis
- Extended test suites,
Martin Quinson <=