an autotest design flaw

[Bruno Haible]

Hi,

Some GNU packages (like GNU cpio) use autotest for their testsuite.

When I'm compiling such a package on a not-so-common platform, and one
of the tests fails, it makes it very hard to investigate the test failure.
The steps to investigate a test failure typically are:
  - Get a list of the shell commands that were executed by the test.
  - Execute these shell commands manually, some of the modified to use a
    debugger.
  - Modify the test to show additional information or to perform different
    shell commands.

Unfortunately, all this is impossible without all development tools
available on the target platform: GNU m4, Perl, GNU autoconf. And porting
Perl can be an adventure by itself...

In practice, when these development tools are not available on the target
platform, the required steps to investigate a test are:
  - Get back to a development machine,
  - Modify the test, regenerate a complete distribution ("make dist"),
  - Transfer it to the target machine, unpack, recompile.

In other words, the concept of autotest seems to be optimized for simply
_running_ the tests in read-only mode, not for _developing_ with the tests.

What could be the fix? IMO one needs to get away from the concept of
generating something at distribution time. I think the right implementation
would be a library of shell functions.

Bruno