Re: [GSoC] Early design discussion for TAP/SubUnit support in automake.

[Stefano Lattarini]

On Sunday 22 May 2011, Ralf Wildenhues wrote:
> Hi Stefano, and sorry for the long delay,
>
No problem, you had warned me in due time about such possible delays this
month; so there's really no need to apologize.

> * Stefano Lattarini wrote on Fri, Apr 29, 2011 at 11:21:06AM CEST:
> > Now that my GSoC application "automake - Interfacing with a test protocol
> > like TAP or subunit" has been officially accepted, I'd like to start
> > discussing with the community some early, high-level design and interface
> > decisions.
> 
> 
> >  1. Reuse parallel-tests "framework"
> >  -----------------------------------
> > 
> >   The new TAP/SubUnit support should reuse as much of the current
> >   parallel-tests implementation and semantics as possible.  In particular,
> >   it should be able to run different test scripts in parallel, generate a
> >   `.log' file for each test script and a "summarizing" `test-suite.log'
> >   file, honour the make variables AM_TESTS_ENVIRONMENT, TESTS_ENVIRONMENT
> >   and AM_COLOR_TESTS and the environment variable VERBOSE, and support
> >   different extensions for the test scripts, with extension-specific "log
> >   compilers" and flags (the stuff enabled by TESTS_EXTENSIONS,
> >   <ext>_LOG_COMPILER, etc.).
> 
> Sounds all sane.
> 
> >   The XFAIL_TESTS variable might be still supported for the sake of
> >   backward-compatibility (see below for the details), but it should be
> >   deprecated, since TAP and SubUnit offer better and more granular ways
> >   to express expected failures.
> 
> OK.
> 
> In another mail:
> > Thinking again about this, it might be worth trying to be even more 
> > consistent
> > with the existing parallel-tests functionality, and use an 
> > `ext_TEST_PROTOCOL'
> > variable (or similar) instead of a global `tests-protocol' option.  With 
> > some
> > tweaking to the post-processing of `.log' files done in `lib/am/check.am' 
> > (to
> > generate `$(TEST_SUITE_LOG)'), this might allow greater code reuse and a 
> > more
> > consistent API.
> > 
> > I've started experimenting with this idea, and I'm not seeing any obvious
> > shortcoming right now.  I'm hoping I'll be able to post some experimental
> > patches soon enough.
> 
> Allowing to specify that per-test is a good idea for transitioning test
> suites.
>
About this, in my first two "tentative" patches:
 <http://lists.gnu.org/archive/html/automake-patches/2011-05/msg00093.html>
I've taken an even more general approach, allowing the developer to define
and use his own program(s) to:
 1. launch his test scripts,
 2. interpreter their results,
 3. display these results on screen, and
 4. format and generate the log files.
All of this is attainable simply by assigning a variable `LOG_WRAPPER'
(and extensione-specific counterparts of it), and, well, obviously
providing a real "driver" script that obeys to minimal command-line
interface (so that it can grasp the options the Automake-generated
Makefiles passes to it).  Then we will hopefully be able to implement
our TAP/SubUnit parsers on the top of this feature (thus making it
indirectly more tested, which is always good for a new feature).

> I hope to look into the posted patches later today.
>
About this, please note that I might be AFK until this evening.  So have
no haste.

> >  2. New automake option `tests-protocol'
> >  ---------------------------------------
> > 
> >  The Tap/SubUnit support in the Automake-generated testsuite drivers
> >  should be enabled by a new (argument-requiring) option `tests-protocol',
> >  that will be used to specify the level of support for, detection of, and
> >  enforcing of SubUnit/TAP streams.
> > 
> >  The possible values for `tests-protocol' will be:
> >   - tests-protocol=tap
> >     All test scripts are expected to use the TAP protocol.
> >   - tests-protocol=subunit
> >     All test scripts are expected to use the SubUnit protocol.
> >   - tests-protocol=adaptive
> 
> The way you describe "adaptive", it sounds like it should rather be
> named something like "detect" or "detected" or so.
>
I'd like to withdraw this proposal now that we can define per-extension test
protocols.  Having our hypotetical "client developer" rename his test scripts
as they get converted to use TAP/SubUnit is IMO better than we having to
implement in Automake a probably non-trivial "metaparser" that could end up
being scarcerly used anyway.  WDYT?

> >     Each test script is expected to print on its first line of output
> >     which protocol it uses (the exact format of this special line is
> >     still to be determined); if this line is unrecognized, the driver
> >     should assume that the test script uses no protocol.  Also, in
> >     this case, we should continue to honour XFAIL_TESTS.  All of this
> >     should help to maximize backward-compatibility.
> 
> 
> 
> >  3. Console output from the test driver
> >  ---------------------------------------
> > 
> >   This output should remain as close as possible to the one already
> >   provided by the current parallel-tests driver.  The following example
> >   should help to clarify what I mean.
> > 
> >   Assume we are using `tests-protocol=adaptive', and let TESTS be defined
> >   to "pass.test skip.test subunit.test tap.test".  Here, `pass.test' and
> >   `skip.test' are test scripts that use no protocol (and that exit with
> >   status `0' and `77' respectively), `subunit.test' is a test script using
> >   the SubUnit protocol and containing four testcases (one passing, one
> >   failing, one skipped and one which incurs in an expected failure), and
> >   `tap.test' is a test script using the TAP protocol which runs two
> >   successful testcases, then encounters an internal error and bails out
> >   (using the "Bail out!" directive).
> > 
> >   With such a setup, this is the output I'd expect from "make check":
> >     PASS: pass.test
> >     SKIP: skip.test
> >     PASS: subunit.test [testcase name/description]
> >     FAIL: subunit.test [testcase name/ description]
> >     SKIP: subunit.test [testcase name/description] [reason for skipping]
> >     XFAIL: subunit.test [testcase name/description] [failure reason]
> >     PASS: tap.test [testcase name/description]
> >     PASS: tap.test [testcase name/description]
> >     ERROR: tap.test [reason for the bailout]
> > 
> >   Of course, the `color-tests' option should make the above output properly
> >   colorized; the attached html file shows what colors I'd expect.
> 
> Wrt. to the details of the output, I wouldn't fix things too early.
> Two things two consider here:
> - if tests are run in parallel, you want to avoid intermixing output
>   from different tests as much as possible,
>
I disagree.  Once the test script name is printed in each line which report
the result of one of its test cases, I see no issue with intermixed lines.
I.e., I don't believe that this:

  PASS: pass.test
  PASS: subunit.test [foo]
  PASS: tap.test [t1]
  SKIP: skip.test
  PASS: tap.test [t2]
  FAIL: subunit.test [bar]
  SKIP: subunit.test [baz] [reason for skipping]
  ERROR: tap.test [reason for the bailout]
  XFAIL: subunit.test [quux] [failure reason]

is any less clear than this:

  PASS: pass.test
  SKIP: skip.test
  PASS: subunit.test [testcase name/description]
  FAIL: subunit.test [testcase name/ description]
  SKIP: subunit.test [testcase name/description] [reason for skipping]
  XFAIL: subunit.test [testcase name/description] [failure reason]
  PASS: tap.test [testcase name/description]
  PASS: tap.test [testcase name/description]
  ERROR: tap.test [reason for the bailout] XXX

>   but even more than that, you want to avoid intermixing within lines.
>
OTOH, I do believe this is a real concern, to be carefully addressed and
tested for.  Thanks for bringing this up.

>   Gathering output and
>   printing it all at once can help here, but the downside is possibly
>   losing output if tests are interrupted, or the driver doesn't finish
>   correctly.
> - some sort of bypass of the (currently hidden) output is helpful,
>   some gnulib-using packages do that already by redirecting stderr
>   before running the test, and printing a reason for skipping from
>   within the test.
>
[thinking about this put on the TODO list]

> I think the basic premise to take into account is that the test driver
> author (you) may not know all requirements a future test author may
> have.
>
That's why I like my current idea of allowing the client developers
to provide their own testsuite driver.

> >  4. RST support and HTML generation: should be dropped?
> >  ------------------------------------------------------
> 
> Good question.  You could do a poll on the automake list (in a separate
> mail & thread, to gain visbility).  Or just defer the implementation
> for later.
> 
> In another mail:
> > BTW, I also hope that new interface I'm planning will make it easier to
> > implement HTML report generation also for TAP and SubUnit tests, with a
> > consistent reuse of the existing code.  In which case my considerations
> > above will become moot.
> 
> Cool.
>
The downside of this is that it might place additional burdens on the
writer of test drivers. Hmmm... maybe a new option "no-html-test" or
similar is warranted?

> Thanks!
> Ralf
> 

Thanks for the review.

Regards,
 Stefano