Re: [GSoC] Early design discussion for TAP/SubUnit support in automake.

[Ralf Wildenhues]

Hi Stefano, and sorry for the long delay,

* Stefano Lattarini wrote on Fri, Apr 29, 2011 at 11:21:06AM CEST:
> Now that my GSoC application "automake - Interfacing with a test protocol
> like TAP or subunit" has been officially accepted, I'd like to start
> discussing with the community some early, high-level design and interface
> decisions.


>  1. Reuse parallel-tests "framework"
>  -----------------------------------
> 
>   The new TAP/SubUnit support should reuse as much of the current
>   parallel-tests implementation and semantics as possible.  In particular,
>   it should be able to run different test scripts in parallel, generate a
>   `.log' file for each test script and a "summarizing" `test-suite.log'
>   file, honour the make variables AM_TESTS_ENVIRONMENT, TESTS_ENVIRONMENT
>   and AM_COLOR_TESTS and the environment variable VERBOSE, and support
>   different extensions for the test scripts, with extension-specific "log
>   compilers" and flags (the stuff enabled by TESTS_EXTENSIONS,
>   <ext>_LOG_COMPILER, etc.).

Sounds all sane.

>   The XFAIL_TESTS variable might be still supported for the sake of
>   backward-compatibility (see below for the details), but it should be
>   deprecated, since TAP and SubUnit offer better and more granular ways
>   to express expected failures.

OK.

In another mail:
> Thinking again about this, it might be worth trying to be even more consistent
> with the existing parallel-tests functionality, and use an `ext_TEST_PROTOCOL'
> variable (or similar) instead of a global `tests-protocol' option.  With some
> tweaking to the post-processing of `.log' files done in `lib/am/check.am' (to
> generate `$(TEST_SUITE_LOG)'), this might allow greater code reuse and a more
> consistent API.
> 
> I've started experimenting with this idea, and I'm not seeing any obvious
> shortcoming right now.  I'm hoping I'll be able to post some experimental
> patches soon enough.

Allowing to specify that per-test is a good idea for transitioning test
suites.  I hope to look into the posted patches later today.

>  2. New automake option `tests-protocol'
>  ---------------------------------------
> 
>  The Tap/SubUnit support in the Automake-generated testsuite drivers
>  should be enabled by a new (argument-requiring) option `tests-protocol',
>  that will be used to specify the level of support for, detection of, and
>  enforcing of SubUnit/TAP streams.
> 
>  The possible values for `tests-protocol' will be:
>   - tests-protocol=tap
>     All test scripts are expected to use the TAP protocol.
>   - tests-protocol=subunit
>     All test scripts are expected to use the SubUnit protocol.
>   - tests-protocol=adaptive

The way you describe "adaptive", it sounds like it should rather be
named something like "detect" or "detected" or so.

>     Each test script is expected to print on its first line of output
>     which protocol it uses (the exact format of this special line is
>     still to be determined); if this line is unrecognized, the driver
>     should assume that the test script uses no protocol.  Also, in
>     this case, we should continue to honour XFAIL_TESTS.  All of this
>     should help to maximize backward-compatibility.



>  3. Console output from the test driver
>  ---------------------------------------
> 
>   This output should remain as close as possible to the one already
>   provided by the current parallel-tests driver.  The following example
>   should help to clarify what I mean.
> 
>   Assume we are using `tests-protocol=adaptive', and let TESTS be defined
>   to "pass.test skip.test subunit.test tap.test".  Here, `pass.test' and
>   `skip.test' are test scripts that use no protocol (and that exit with
>   status `0' and `77' respectively), `subunit.test' is a test script using
>   the SubUnit protocol and containing four testcases (one passing, one
>   failing, one skipped and one which incurs in an expected failure), and
>   `tap.test' is a test script using the TAP protocol which runs two
>   successful testcases, then encounters an internal error and bails out
>   (using the "Bail out!" directive).
> 
>   With such a setup, this is the output I'd expect from "make check":
>     PASS: pass.test
>     SKIP: skip.test
>     PASS: subunit.test [testcase name/description]
>     FAIL: subunit.test [testcase name/ description]
>     SKIP: subunit.test [testcase name/description] [reason for skipping]
>     XFAIL: subunit.test [testcase name/description] [failure reason]
>     PASS: tap.test [testcase name/description]
>     PASS: tap.test [testcase name/description]
>     ERROR: tap.test [reason for the bailout]
> 
>   Of course, the `color-tests' option should make the above output properly
>   colorized; the attached html file shows what colors I'd expect.

Wrt. to the details of the output, I wouldn't fix things too early.
Two things two consider here:
- if tests are run in parallel, you want to avoid intermixing output
  from different tests as much as possible, but even more than that,
  you want to avoid intermixing within lines.  Gathering output and
  printing it all at once can help here, but the downside is possibly
  losing output if tests are interrupted, or the driver doesn't finish
  correctly.
- some sort of bypass of the (currently hidden) output is helpful,
  some gnulib-using packages do that already by redirecting stderr
  before running the test, and printing a reason for skipping from
  within the test.

I think the basic premise to take into account is that the test driver
author (you) may not know all requirements a future test author may
have.

>  4. RST support and HTML generation: should be dropped?
>  ------------------------------------------------------

Good question.  You could do a poll on the automake list (in a separate
mail & thread, to gain visbility).  Or just defer the implementation
for later.

In another mail:
> BTW, I also hope that new interface I'm planning will make it easier to
> implement HTML report generation also for TAP and SubUnit tests, with a
> consistent reuse of the existing code.  In which case my considerations
> above will become moot.

Cool.

Thanks!
Ralf