Re: Various testsuites

[Ralf Wildenhues]

Hello,

* NightStrike wrote on Mon, May 18, 2009 at 09:41:27PM CEST:
> What's the difference between using this:
> http://www.gnu.org/software/autoconf/manual/autoconf.html#Using-Autotest
> 
> and this:
> http://www.gnu.org/software/automake/manual/automake.html#Tests
> 
> ?

Autotest is a generated portable shell script, similar to configure,
that you can, ideally, distribute stand-alone, and built with a set of
m4 macros like configure.ac is (but a slightly different set).

Automake TESTS support, as of version 1.11, consists of two different
test drivers that run tests from within a makefile.  The old one is
pretty minimalistic in test suite functionality, the new one (selected
with the `parallel-tests' option) is mostly backward-compatible, but
also offers some more and slightly changed functionality.  It still
operates as a set of rules within a makefile.

DejaGNU (which you didn't mention) is a separate package that provides
yet another, quite elaborate, test suite framework.  It is quite
powerful, but may be overkill for simpler uses.  YMMV.

> Which is better to use?

Good question.  The general answer is "it depends", I would say,

> Which will be maintained,

It is intended that all of them be maintained.

> and more future-proof?

I cannot answer that well, really.  I do hope though that
future-proofness should not be a deciding point.

> Why is there duplication between two products which are very
> inter-related?

Because these test suite frameworks grew separately, and with slightly
different purposes, and slightly different strengths and weaknesses.

For example, DejaGNU works very well with complex setups where a program
(say, a compiler) needs to be run on many different files, as in the GCC
test suite.  Being Tcl, is way more efficient than a bunch of shell
scripts with lots of forks.

OTOH, Autotest and Automake TESTS have less software requirements.

Autotest has quite some overhead due to aiming to be in portable shell.
The test structure is a bit more rigid when compared to Automake TESTS,
but also initialization and cleanup handling of test group directories
is all built-in and requires practically no developer intervention.
At the moment, there is experimental support for parallel execution on
some systems.

Automake TESTS has little initialization and cleanup handling, that
typically needs to be open-coded by the developer.  The parallel-tests
driver supports efficient parallel test execution through `make' level
parallelism, and it supports test wrappers based on file name extension
and such.  It is not as shell-centered as Autotest.

All frameworks have some level of summary reports.

It really depends upon the kind of testing that you would like to do,
and I would choose different of these frameworks based on the task at
hand.  I've also omitted lots of features in this message.

Cheers,
Ralf