Re: [GSoC Proposal] automake - Interfacing with a test protocol like TAP or subunit

[Stefano Lattarini]

Hello Robert and Jelmer, and thanks for your replies.

On Tuesday 22 March 2011, Jelmer Vernooij wrote:
> On Tue, 2011-03-22 at 10:48 +1300, Robert Collins wrote:
> > On Tue, Mar 22, 2011 at 8:41 AM, Stefano Lattarini
> > > to its suboptimal documentation.  So I'm going to ask: Robert, as
> > > the main proposer/supporter of the SubUnit protocol here, would you
> > > be willing and ready to help me out during my prospective work with
> > > GSoC, if I update my application's goal to read "Support SubUnit
> > > (and also TAP as derivation) in Automake-generated testsuites"?
> > > Maybe we could also improve SubUnit's documentation along the way,
> > > which would help both me and Automake, and also improve SubUnit
> > > itself and make it more "palatable" for potential adopters.
> > I'll be delighted to help by clarifying things or making sensible
> > small improvements. We should structure things so there is a clear API
> > between whatever automake needs and subunit providing it (this allows
> > subunit to continue to improve without automake needing to change
> > futher, and vice versa). I would offer to make large changes if
> > needed, but realistically, work is in the middle of a crucial project
> > that is going to be going on for ~ 3-4 more months, so I am somewhat
> > time constrained :). That said, Jelmer Vernooij may well - he has
> > written a number of Perl subunit tools in the samba project, and they
> > are likely to be at least somewhat relevant to whatever automake needs
> > are here.
> I'd be happy to help with inquiries about subunit where I can, or
> co-mentor, whatever works best. 
>
The former would already be great and very helpful -- thanks!

About the latter, that would be great too, but I guess I'm in no position
to vouch for it, or to judge if it's really required.  Anyway, I see
you've already been a GSoC mentor for four years now, so I guess you
are able to decide what is necessary or advisable here, and in case,
undertake the necessary steps.  I'll defer to your judgement in this
matter.

> FWIW the Perl subunit modules shipped with subunit and included in Samba
> are sufficient for parsing existing subunit streams and for generating
> subunit streams, but they're still pretty basic. They don't yet
> integrate with any of the standard perl testing frameworks like
> Test::Simple or Test::More and they (currently) lack support for some of
> the newer subunit features.
>
Thanks for the information.  Still, I'll only be able to take clues and/or
"inspiration" from them: their direct use is not viable because
Automake-generated Makefiles cannot assume the presence of Perl; the best
we can do is go with shell + sed + awk ...

In the meantime, I've updated my application and roadmap to reflect the
new SubUnit focus.  Find them attached if you're interested.

Thanks again,
   Stefano

Student Identification
======================

 Name: Stefano Lattarini
 Email address: address@hidden


The name of the project
=======================

 automake - Interfacing with the test protocols TAP and SubUnit


Abstract
========

 The Test Anything Protocol (TAP) and the SubUnit protocol are simple
 text-based protocols that allow communication between test scripts and
 a test harness.

 The primary target of this project is the implementation of a TAP-consumer
 and a SubUnit-consumer testsuite harness in Automake-generated Makefiles.
 This way, a software package using Automake and having a scripts-based
 testsuite would be able to run multiple checks in each test script, and
 communicate the result of each of those checks separately to the test
 harness, which could then issue more faithful statistics about the number
 of failed/passed/skipped checks.

 A secondary objective, which will be pursued only if the completion of
 the primary one will leave enough time, is to enhance Autotest-generated
 test scripts to allow them to become TAP and/or SubUnit producers.


The parallel-tests driver and its limits
========================================

 In the recent years, Automake's built-in support for the generation
 of testsuite harnesses has become quite sophisticated.  In particular,
 the "parallel-tests" driver has performed very good.

 That driver is targeted at testsuites comprised of test scripts which
 are mostly independent one from the other, and safe to run in parallel.
 It is implemented in portable make (obviously with the contribution of
 portable POSIX utilities, such as the shell, awk, sed, etc.), but still
 manages to offer many useful and pretty advanced features (please refer
 to the Automake documentation for more details).

 Still, the parallel driver still has one major limit.

 In a scripts-based testsuite, it's common for a single test script
 to run checks for multiple (usually related) features, and/or against
 multiple bugs at once -- especially in case these checks require a
 similar (or even identical) setup.  And while it would usually be
 feasible to separate such related checks into different test scripts,
 that might require a too-high overhead, either in test execution time
 or in programmer coding time.

 The problem with the current Automake parallel-tests driver (and also
 with its precursor, the "simple-tests" driver) is that the results of
 all the checks run by an individual test script are hopelessly condensed
 into the single final exit status of the script.  So there is no way
 to distinguish between, say, a test script that failed because one check
 out of N in it failed, from one which failed because all the N checks
 in it failed.


Possible enhancements with the TAP and SubUnit protocols
========================================================

 The Test Anything Protocol (TAP) and the SubUnit protocol are simple
 text-based protocols that allow communication between test scripts and
 a test harness.  They allow individual tests (TAP/SubUnit producers) to
 communicate test results to the testing harness (TAP/SubUnit consumer)
 in a language-agnostic way.  What is important for us is that, using
 one of these protocols, a single test script can safely run multiple
 checks and report the outcome of each of them separately to the harness;
 the results will then be properly displayed and accounted for by the
 harness, with the right level of granularity.

 So, for example, if we have a properly written TAP-producing test
 script that fails because two checks out of 20 in it failed, the test
 harness won't count that as "1 failure, 0 passes", but as "2 failures,
 18 passes".

 Thus, Automake-generated testsuite harnesses could become TAP and/or
 SubUnit consumers, for third-party testsuites and for Automake-generated
 parallel testsuites alike.

 Also, SubUnit can be seen a "superset" of TAP (while *not* being a TAP
 extension in a strict sense), and it's possible to convert TAP into
 SubUnit without too much difficulty (the SubUnit distribution offers a
 simple python class that is able to do so).  Thus, if we manage to make
 Automake-generated test harnesses able to parse SubUnit, the insertion
 of a simple filter acting like a TAP->SubUnit converter will almost
 automatically make them able to understand TAP too.

 Finally, Autoconf's Autotest currently use its own custom output to
 report test results.  It could benefit of better interoperability if
 it starts supporting TAP and/or Subunit; in particular, the integration
 with the Automake-generated harnesses would be greatly improved (and
 probably also simplified).

 Some useful links on TAP:
  * <http://search.cpan.org/~petdance/Test-Harness/lib/Test/Harness/TAP.pod>
    Documentation for the TAP format understood by the perl module
    Test::Harness (kind of "de-facto standard");
  * <http://testanything.org/wiki/index.php/Main_Page>
    A wiki on TAP, with useful discussions, pointers, and also bits
    of history.
  * <http://www.eyrie.org/~eagle/software/c-tap-harness/>
    A C library implementing both a TAP producer and a TAP consumer;
    should be quite mature and actively maintained.

 The SubUnit homepage:
  * <https://launchpad.net/subunit>
 contains some basic documentation on SubUnit, and SubUnit producers
 for various languages (python, perl, C and C++).  The Samba project
 is an early adopter of SubUnit, using it in (at least part of) its
 testsuite:
  * <http://jelmer.vernstok.nl/blog/archives/262-subunit-usage-in-Samba.html>
  * <http://samba.org/~jelmer/samba4-testing.pdf>


Benefits of this proposal
=========================

 We have already clearly described how the use of TAP/SubUnit would
 improve the granularity in the presentation/summary of test results.

 The TAP/SubUnit model also offers what I deem to be a better mental
 model of the typical "testing with test scripts" scenario: a testsuite
 contains test scripts, each of which can run multiple checks.  The
 smallest meaningful piece of result is the outcome of a check, and
 *not* the exit status of a test script.  This model is both more
 granular and more natural.

 TAP is well-established in the Perl community, and thus the protocol
 have already proven itself as valid "in the real world".  This seems
 confirmed by the existence of quite many third-party TAP producers
 and consumers for different languages.

 SubUnit is admittedly not yet well-established as that; still, being
 designed with TAP compatibility in mind, it should share many TAP
 qualities.  One advantage it has over TAP is that it's meant to work
 well with more modern programming languages (e.g., python) and more
 widespread testing paradigms (xUnit).  Last but not least, it is
 already used by at least an important real-word project like Samba.

 There are already C/C++ libraries aimed at writing TAP-producing and/or
 SubUnit-producing test programs.  Thus the addition of the new TAP
 and/or Subunit support in Automake would allow an easier and more
 proficient integration of custom C unit tests into Automake-generated
 testsuite harnesses.  This would be particularly useful, considering
 that projects based on C and/or C++ are the primary client base of
 Automake.

 Finally, the TAP/SubUnit support in Automake could rely on the existing
 implementation and APIs of the parallel-tests driver, extending them
 in a mostly (if not even complete) backward-compatible way.  The new
 functionality would thus build on a well-established foundation, and
 place itself in a thriving paradigm.


Deliverables
============

 Once this proposal is implemented, Automake should be able to generate
 TAP-consumer and SubUnit-consumer testsuite harnesses that work portably
 on many systems (ideally all those Automake strives to support) and with
 many vendor make implementations (ideally all those Automake strives to
 support).  Failing this is due time, the "fallback plan" is to have the
 generated testsuite harnesses either:
  - support only one among SubUnit and TAP; or
  - support both, but with the additional requirement of perl (>= 5.6
    should be enough) being installed on the target system;

 Finally, if there is enough time left, Autotest might be enhanced so
 that the test scripts it generates become able to produce TAP and/or
 SubUnit output.  This is not a primary target, though, and might as
 well dropped if it's clear there won't be enough time to complete it.


Plan and Roadmap
================

 Q: How will you and your mentor track your progress as you work on the
    project?
 A: Since technical discussions about design and implementation of new
    Automake features take place on the public lists <address@hidden>
    and/or <address@hidden>, most measurable progresses are
    publicly available there, and thus quite easy to judge.  Also, every
    new feature going in Automake must have proper testsuite coverage,
    so that we can be confident that the new code will really works.
    Finally, the project mentor has been the Automake maintainer since
    4/5 years now, so that he's surely able to judge how a partial
    implementation if faring, and what the chances are for it to be
    completed in time.

 Q: How the mid-term evaluation of your project will be made?
 A: I must have an implementation of Automake-generated TAP-consumer
    testsuite harness that works at least with non-obsolete versions of
    GNU make at least on GNU/Linux, Solaris 10 and FreeBSD 8.  This
    implementation must comprise test cases (maybe non exhaustive) and
    documentation (also  unpolished, but basically complete).

 Q: Remember to mention any periods during the summer when you won't
    actually be available to work on the project (though remember, the
    Summer of Code project is expected to be your main activity).
 A: I plan to be available for basically all the period of the project.
    If not the whole day, at least half a day, basically each day (either
    morning and early afternoon, or later afternoon and evening).  There
    might be *very few* short periods (one, max two days) when I'll be
    completely off-line and "away from keyboard" -- I'll let the mentor
    know about them in advance, obviously.


Communication
=============

  First, I'm already in contact with the mentor, due to my on-going
  involvement with the Automake project.  I've also been in contact
  with the backup mentor, due to my (small) contributions to Autoconf.

  In the Automake and Autoconf projects, discussions about design and
  documentation, and proposals and reviews of patches are public, taking
  place on the projects' pre-existing and well-established mailing lists.
  The mentor reads them regularly, so that posting patches an ideas there
  is a sure way to have them considered and reviewed by him (and possibly
  also by other mailing list readers, which might contribute with their
  own useful insights, questions, and experiences).

  Finally, my experiences have showed that real-time interaction with
  the mentor to work on a particular issue (either on-list or off-list)
  is not only possible, but also efficient and effective.


Qualification
=============

 I'm quite familiar with the policies of both the Automake and Autoconf
 projects (especially coding standards, patch review process, and
 portability concerns).  Moreover, the copyright assignment to the Free
 Software Foundation that allows me to contribute code to the Automake
 and Autoconf official repositories is already in place (and has been
 for more than a year and a half now).

 I've started contributing to the Automake project an year and a half
 ago.  During the last year, my involvement with the project has steadily
 increased: I've posted more patches and participated more actively to
 discussions on the mailing lists, I've been granted commit rights, and
 I've started helping with the bug tracker and related issues.  I plan
 to continue my involvement with Automake in the foreseeable future.

 I have access to some systems to test my work on: Debian GNU/Linux
 (with custom installation of older versions of tools and compilers),
 FreeBSD 8.0, and Solaris 10.  This can help me in finding out early
 potential portability problems.

 I believe that powerful, easy-to-use testing frameworks are vital for
 the development of modern software.  Also, I find software testing an
 interesting, and sometimes even fascinating, topic.  So I can say I'm
 motivated enough to work in this area.

Tentative roadmap for GSoC project "automake - Interfacing with a test
protocol like TAP or subunit".


1. COLLECT INFORMATION: Before April 20
----------------------------------------

 Read carefully the TAP and SubUnit specifications available.  Since the
 SubUnit specification is somewhat lacking, in case of doubts/problems
 ask to the authors/supporters (they've already proven themselves willing
 to help in this regard).

 Take a look at real-word usages of TAP (mostly the distribution of Perl
 and some major perl packages) and SubUnit (mostly the Samba testsuite);
 familiarize as much as possible with the corner cases and the "rough
 edges", jot down some notes about things that are unclear might be
 deemed to cause potential problem.

 Finally, if there's still time, take a look at existing implementations;
 chiefly:
  - for TAP, Test::Harness and c-tap-harness; and
  - for SubUnit, the subunit package and some support code in the Samba
    testsuite.
 All the licenses involved are GPL-compatible, so this should be OK from
 a legal point of view.


2. EXPERIMENTING: April 20 - May 15  (Before the official coding time)
-----------------------------------------------------------------------

 Write scripts (better if in pure awk, for performance reasons; but shell
 scripts are acceptable) and Makefiles that implement TAP and/or SubUnit
 consumers (it's OK to do so by tweaking Automake-generated Makefiles,
 and it's ok if they're not 100% functional in this phase).

 Try to make those examples as portable as possible, to spot potential
 portability issues early.  Test at least on GNU/Linux, FreeBSD 8 and
 Solaris 10, and their native make, shell and awk implementation(s).

 Exchange early ideas and doubts with the mentor, and, if appropriate,
 also with the official mailing lists.

 If there's still time, take a deeper look at Autotest documentation,
 implementation, and history, as to be able to later (try to) extend
 it to produce TAP and/or SubUnit compatible output (time permitting).

 This phase should help with the subsequent "real" implementation, by
 providing some ideas, and quite likely also code (incomplete of course,
 but probably good as a starting point and/or for "cut & paste"), and
 tentative testcases.


3. BASIC DESIGN: May 15 - June 5  (Official coding period starts May 23)
-------------------------------------------------------------------------

 We'll have to decide which is the best way to integrate the TAP/SubUnit
 consumer code into Automake -- will we use new primary, a new option,
 or maybe some new special syntax and/or variables?  This is something
 to be discussed publicly on the Automake mailing lists.

 Also, how much of the parallel-tests implementation and scaffolding
 can be reused?  What about parallelization and/or synchronization
 issues?  What about other issues that came up in the experimentation
 phase?  All these doubts should be worked out.

 If there's time, we might also start planning how Autotest can be
 extended to allow it to produce TAP/SubUnit output.  This is something
 to be discussed publicly on the Autoconf mailing lists.

 BTW, at this point, we'll probably need to go back and do some more
 experimenting (but more purpose-oriented than in the previous phase).
 And we could also write some early test cases and documentation.


4. WRITE BASIC TESTCASES AND DOCUMENTATION: June 5 - June 12
-------------------------------------------------------------

 Documentation about design and general concepts shouldn't be long
 nor verbose, but it's important to get it right and to make it as
 clear as possible -- which can be tricky, and take some time.  Also,
 condensing the most important use cases into tests is not immediate
 (although it shouldn't be very difficult in this phase).


5. CODING, PART 1: June 12 - July 6  (Until mid-term evaluation)
----------------------------------------------------------------

 Here I'll have to extend Automake to provide TAP and SubUnit aware test
 harnesses.  The resulting code must behave as to match the existing
 documentation and design.  It should also be able to generate Makefiles
 that work on at least GNU/Linux, FreeBSD and Solaris, with their native
 make, shell and awk implementation(s).  In case I fail to accomplish
 this in due time, here are my fallback plans:
  - support only one among TAP and SubUnit -- but support it portably!
 or:
  - support both TAP and SubUnit, not fully portably -- e.g., relying
    on a Perl interpreter to be installed on the target system.

 This is not a clear-cut phase.  During it, I'll have to be ready to
 extend and adjust the new documentation; adapt the existing testcases
 and add some more (probably many more); read and understand parts of
 the Automake codebase I'm not familiar with yet; and maybe doing more
 experimentation.  In the very worst case, the mentor and I might even
 be forced to revisit and modify our previous design.


5. CODING, PART 2: July 6 - July 23
------------------------------------

 If the implementation for TAP and SubUnit aware test harnesses written
 in the previous phase is complete and wholly functional (which include
 portability to at least BSD and Solaris make, and independence from
 Perl, but might allow for some rough edges), in this phase I'll try to
 enhance Autotest to enable TAP and/or SubUnit compatible output in its
 generated test scripts.  Otherwise, I'll have to work more on Automake,
 ditching the plans for Autotest enhancements.


6. DOCUMENTATION POLISHING: July 23 - July 28
----------------------------------------------

 Polish/complete documentation of all the implemented features.  Maybe
 add some examples, and if there's time contribute them to existing
 third-party tutorials, FAQs and/or wikis about the autotools.


7. STRESS AND PORTABILITY TESTING: July 28 - Until done
--------------------------------------------------------

 Test the code on other systems which Automake strives to support (e.g.,
 Cygwin, MSYS/MinGW, AIX, HP-UX, IRIX), and (try to) fix all the problems
 and incompatibilities that show up.  This might require some more help
 from the mentor, which has access to an impressive array of testing
 systems (all the major proprietary Unix implementations are among them).


8. FINAL CLEANUP: All the time that's left (if any)
----------------------------------------------------

 Note that this last pass is not strictly necessary, and might be omitted
 altogether in case we're out of time.

 Here we could add more tests about corner cases; polish documentation
 and/or code; and maybe even convert (part of) Automake's own testsuite
 to the use of the newly implemented TAP/SubUnit-based harness.