[GSoC] Early design discussion for TAP/SubUnit support in automake.

[Stefano Lattarini]

Hello automakers.

Now that my GSoC application "automake - Interfacing with a test protocol
like TAP or subunit" has been officially accepted, I'd like to start
discussing with the community some early, high-level design and interface
decisions.

JFTR, you can read my GSoC application here:
 
<http://www.google-melange.com/gsoc/proposal/review/google/gsoc2011/slattarini/1>

And here are some useful links where you can find more information about
TAP and SubUnit:

 - The documentation for the TAP format understood by the perl module
   `Test::Harness' (kind of "de-facto standard"):
   <http://search.cpan.org/~petdance/Test-Harness/lib/Test/Harness/TAP.pod>

 - A wiki on the TAP protocol and format:
   <http://testanything.org/wiki/index.php/Main_Page>

 - A C-based project implementing both a TAP producer and a TAP consumer:
   <http://www.eyrie.org/~eagle/software/c-tap-harness/>

 - The SubUnit homepage:
   <https://launchpad.net/subunit>

 - A  blog entry about SubUnit usage in Samba:
   <http://jelmer.vernstok.nl/blog/archives/262-subunit-usage-in-Samba.html>


First Tentative Decisions and Questions on the User Interface
=============================================================


 1. Reuse parallel-tests "framework"
 -----------------------------------

  The new TAP/SubUnit support should reuse as much of the current
  parallel-tests implementation and semantics as possible.  In particular,
  it should be able to run different test scripts in parallel, generate a
  `.log' file for each test script and a "summarizing" `test-suite.log'
  file, honour the make variables AM_TESTS_ENVIRONMENT, TESTS_ENVIRONMENT
  and AM_COLOR_TESTS and the environment variable VERBOSE, and support
  different extensions for the test scripts, with extension-specific "log
  compilers" and flags (the stuff enabled by TESTS_EXTENSIONS,
  <ext>_LOG_COMPILER, etc.).

  The XFAIL_TESTS variable might be still supported for the sake of
  backward-compatibility (see below for the details), but it should be
  deprecated, since TAP and SubUnit offer better and more granular ways
  to express expected failures.


 2. New automake option `tests-protocol'
 ---------------------------------------

 The Tap/SubUnit support in the Automake-generated testsuite drivers
 should be enabled by a new (argument-requiring) option `tests-protocol',
 that will be used to specify the level of support for, detection of, and
 enforcing of SubUnit/TAP streams.

 The possible values for `tests-protocol' will be:
  - tests-protocol=tap
    All test scripts are expected to use the TAP protocol.
  - tests-protocol=subunit
    All test scripts are expected to use the SubUnit protocol.
  - tests-protocol=adaptive
    Each test script is expected to print on its first line of output
    which protocol it uses (the exact format of this special line is
    still to be determined); if this line is unrecognized, the driver
    should assume that the test script uses no protocol.  Also, in
    this case, we should continue to honour XFAIL_TESTS.  All of this
    should help to maximize backward-compatibility.


 3. Console output from the test driver
 ---------------------------------------

  This output should remain as close as possible to the one already
  provided by the current parallel-tests driver.  The following example
  should help to clarify what I mean.

  Assume we are using `tests-protocol=adaptive', and let TESTS be defined
  to "pass.test skip.test subunit.test tap.test".  Here, `pass.test' and
  `skip.test' are test scripts that use no protocol (and that exit with
  status `0' and `77' respectively), `subunit.test' is a test script using
  the SubUnit protocol and containing four testcases (one passing, one
  failing, one skipped and one which incurs in an expected failure), and
  `tap.test' is a test script using the TAP protocol which runs two
  successful testcases, then encounters an internal error and bails out
  (using the "Bail out!" directive).

  With such a setup, this is the output I'd expect from "make check":
    PASS: pass.test
    SKIP: skip.test
    PASS: subunit.test [testcase name/description]
    FAIL: subunit.test [testcase name/ description]
    SKIP: subunit.test [testcase name/description] [reason for skipping]
    XFAIL: subunit.test [testcase name/description] [failure reason]
    PASS: tap.test [testcase name/description]
    PASS: tap.test [testcase name/description]
    ERROR: tap.test [reason for the bailout]

  Of course, the `color-tests' option should make the above output properly
  colorized; the attached html file shows what colors I'd expect.


 4. RST support and HTML generation: should be dropped?
 ------------------------------------------------------

  I haven't really looked into this yet, so I don't know how much additional
  work (if any!) would be to continue to support generation of HTML reports
  from the testsuite-generated `.log' files in the new TAP/SubUnit testsuite
  driver.  Still, one question that bugs me is: how much is this HTML report
  generation used in Real Life?  Is it really worth to support it out of the
  box, or could we leave it to the interested used to write their own rules
  (maybe in a reusable `*.am' fragment)?


-*-*-

Comments, ideas and criticism are welcome!

Regards,
  Stefano