Re: Making the check target more verbose

[Ralf Wildenhues]

Hello Martin,

* Martin Quinson wrote on Thu, Sep 13, 2007 at 11:16:47AM CEST:
> Hello, you'll find in attachement a patch against the stable 1.10 to make the
> check target a bit more verbose by displaying a summary of the failed tests 
> (or
> xpassed, or xfailled, or skipped).

Thanks for the patch.  There are some issues I have with it:

First, a minor nit, I would not want to see a summary of expected
failures.  The developer knows which tests are expected to fail --
those are listed in the Makefile.am, and the failures are not a sign
of an error, so there's no reason to list them but not list passed
tests.

Second, I have a more fundamental issue with this, and that is one
of general usability.  For example, Automake's testsuite itself has
several hundred tests.  If most of them fail due to some issue, then
again the user will see at the end of a test run only a list of test
names, but not the summary, nor will she know what that list contains
(FAILs, XPASSes, all tests?), because the respective heading may
already have gone out of view.

So, the summary should go at the end, or be repeated at the end.
However, repetition is ugly, at least for small test suites.

But also, if the list of failures is long, then you still cannot see
easily what a list of test contains, FAILs or XPASSes.  So you end up
repeating `FAIL:' at the beginning of the line, or, paginate output or
so.  But then I'd say you're just about at the point where you want to
write a post-processor for the test output.  You can then continue to
use the current Automake code; we could think about integrating the
post-processor into Automake, but please not in the Makefile.

Thirdly, there are test suites where order of output is significant:
For example, the old Libtool testsuite runs 'demo-make.test' several
times, after one of demo-static.test, demo-shared.test or others; and
order matters: typically, the 'make' stage fails only with some
specific test before it.  Listing out of order destroys this
information.

> This becomes very usefull when applying the "one makefile to rule them
> all" philosophy to the tests, since you end up with a awfull lot of
> output, and finding which are the failed test may turn quite difficult.
> 
> On the other hand, that's not completely perfect since it does not work for
> subdirectories,

This is another reason against the patch.

> but I'm still unsure of how it would be possible without using
> some sort of log file (and even so, that's still challenging I think).

Log file(s) will be needed anyway when considering the proposed changes
for parallel testsuite execution.  (Note parallel execution will
likely require that the developer explicitly turn it on, to confirm
that the tests are properly written for it.)

> I hope that you will find it interesting and consider for inclusion in
> the next version of automake.

I hope I've made clear a couple of arguments against the patch in this
form.  ;-)

Cheers,
Ralf