verbosity of test failure feedback (was: Automake 1.11.1 glitch?)

[Ralf Wildenhues]

* Stefano Lattarini wrote on Tue, Sep 07, 2010 at 06:00:04PM CEST:
> Darn, I've been sloppy: I should have also asked you to provide
> us with the (tarred) content of the temporary test directory 
> `tests/silent5.dir'.  Could you kindly send this over to the list
> too?
> 
> (<NOTE> to self and to Ralf: it would be nice if the Automake 
> testsuite could create, whenever a failure occurs, a tarball containg 
> all the relevant logs and the contents of all relevant temporary test
> directories, to avoid plaguing the poor bug reporters with continuos 
> requests of more details </NOTE>)

Actually, I haven't come across many test failures where more
information was necessary.  At least in this case I think it was
crystal clear from the silent5.log file, no?  ;-)

One practical problem with sending test directory contents by default
is that they can be pretty large.  When multiple failures occur, say,
all Libtool-related tests fail due to a Libtool installation bug, we're
speaking of several MB packed.  I don't really want this sent to a
public mailing list read by dozens of people.

Autotest is better in this regard, in that it allows more fine-grained
logging in case of failure.  Of course we can do it, too, with some more
manual extra work in the tests; still, this work might be more than the
occasional extra question for a user.

Cheers,
Ralf