[GSoC Proposal] automake - Interfacing with a test protocol like TAP or subunit

[Stefano Lattarini]

ABSTRACT:

  The Test Anything Protocol (TAP) is a simple text-based protocol
  that allows communication between test scripts and a test harness.

  The primary target of this project is the implementation of a TAP-consumer
  testsuite harness in Automake-generated Makefiles.  This way, a software
  package using Automake and having a scripts-based testsuite would be able
  to run multiple checks in each test script, and communicate the result of
  each of those checks separately to the test harness, which could then
  issue more faithful statistics about the number of failed/passed/skipped
  checks.

  A secondary objective is to enhance Autotest-generated test scripts to
  allow them to become TAP producers.

DETAILS:

  See attachements.

-*-*-

Now, in all honesty, I must say that I've chosen TAP not only for its
objective qualities and merits, but also because I have some previous
knowledge of it (which allowed me to present a more concrete proposal
and a meaningful roadmap) and I personally like it (which will probably
be a powerful motivator to overcome the unavoidable hurdles I'll
encounter down the road).

Still, there might be very valid competing alternatives to TAP out there,
which I might not know about, but that in the long run would offer
Automake more advantages and interoperabilty, thus outweighting the two
"personal" advantages of TAP I've reported above.  So, if anyone who's
reading this has    proposals about viable alternatives to TAP, please
speak up -- your contribution is appreciated!

Thanks,
  Stefano

Student Identification
======================

 Name: Stefano Lattarini
 Email address: address@hidden


The name of the project
=======================

 automake - Interfacing with a test protocol like TAP or subunit


Abstract
========

 The Test Anything Protocol (TAP) is a simple text-based protocol
 that allows communication between test scripts and a test harness.

 The primary target of this project is the implementation of a TAP-consumer
 testsuite harness in Automake-generated Makefiles.  This way, a software
 package using Automake and having a scripts-based testsuite would be able
 to run multiple checks in each test script, and communicate the result of
 each of those checks separately to the test harness, which could then
 issue more faithful statistics about the number of failed/passed/skipped
 checks.

 A secondary objective is to enhance Autotest-generated test scripts to
 allow them to become TAP producers.


The parallel-tests driver and its limits
========================================

 In the recent years, Automake's built-in support for the generation
 of testsuite harnesses has become quite sophisticated.  In particular,
 the "parallel-tests" driver has performed very good.

 That driver is targeted at testsuites comprised of test scripts which
 are mostly independent one from the other, and safe to run in parallel.
 It is implemented in portable make (obviously with the contribution of
 portable POSIX utilities, such as the shell, awk, sed, etc.), but still
 manages to offer many useful and pretty advanced features (please refer
 to the Automake documentation for more details).

 Still, the parallel driver still has one major limit.

 In a scripts-based testsuite, it's common for a single test script
 to run checks for multiple (usually related) features, and/or against
 multiple bugs at once -- especially in case these checks require a
 similar (or even identical) setup.  And while it would usually be
 feasible to separate such related checks into different test scripts,
 that might require a too-high overhead, either in test execution time
 or in programmer coding time.

 The problem with the current Automake parallel-tests driver (and also
 with its precursor, the "simple-tests" driver) is that the results of
 all the checks run by an individual test script are hopelessly condensed
 into the single final exit status of the script.  So there is no way
 to distinguish between, say, a test script that failed because one check
 out of N in it failed, from one which failed because all the N checks
 in it failed.


Possible enhancements with the TAP protocol
============================================

 The Test Anything Protocol (TAP) is a simple protocol that allows
 communication between test scripts and a test harness.  It allows
 individual tests (TAP producers) to communicate test results to the
 testing harness (TAP consumer) in a language-agnostic way.  What is
 important for us is that, using TAP, a single test script can safely
 run multiple checks and report the outcome of each of them separately
 to the harness; the results will then be properly displayed and
 accounted for by the harness, with the right level of granularity.

 So, for example, if we have a TAP-aware test script that fails because
 one check out of 20 in it failed, the test harness won't count that as
 "1 failure, 0 passes", but as a "1 failure, 19 passes".

 Thus, Automake-generated testsuite harnesses could become TAP consumers,
 for either third-party testsuites and for Automake-generated parallel
 testsuite drivers.

 Also, Autoconf's Autotest currently use its own custom output to report
 test results.  It could benefit of better interoperability if it start
 supporting TAP; in particular, the integration with Automake harnesses
 would be greatly improved (and probably also simplified).

 Some useful links on TAP:
  * <http://search.cpan.org/~petdance/Test-Harness/lib/Test/Harness/TAP.pod>
    Documentation for the TAP format understood by the perl module
    Test::Harness (kind of "de-facto standard");
  * <http://testanything.org/wiki/index.php/Main_Page>
    A wiki on TAP, with useful discussions, pointers, and also bits
    of history.
  * <http://www.eyrie.org/~eagle/software/c-tap-harness/>
    A C library implementing both a TAP producer and a TAP consumer;
    should be quite mature and actively maintained.


Benefits of this proposal
=========================

 We have already clearly described how the use of TAP would an improved
 granularity in the presentation/summary of test results.

 The TAP model also offers what I deem to be a better mental model of
 the typical "testing with test scripts" scenario: a testsuite contains
 test scripts, each of which can run multiple checks.  The smallest
 meaningful piece of result is the outcome of a check, and *not* the
 exit status of a test script.  This model is both more granular and
 more natural.

 TAP is well-established in the Perl community, and thus the protocol
 have already proven itself as valid "in the real world".  This seems
 confirmed by the existence of quite many third-party TAP producers
 and consumers for different languages.

 There are already C/C++ libraries aimed at writing TAP-producing test
 programs.  Thus the addition of the new TAP support in Automake would
 allow an easier and more proficient integration of custom C unit tests
 into Automake-generated testsuite harnesses.  This would be particularly
 useful, considering that projects based on C and/or C++ are the primary
 client base of Automake.

 Finally, the TAP support in Automake could rely on the existing
 implementation and APIs of the parallel-tests driver, extending them
 in a mostly backward-compatible way.  The new functionality would
 thus build on a well-established foundation, and place itself in
 thriving paradigm.


Deliverables
============

 Once this proposal is implemented, Automake should be able to generate
 TAP-consumer testsuite harnesses that works portably on many systems
 (ideally all those Automake strives to support) and with many vendor
 make implementations (ideally all those Automake strives to support).
 Failing this, the "fallback plan" is to have the generated testsuite
 harnesses to work at least with recent and older GNU make version on
 as many systems as possible (ideally all those Automake strives to
 support).

 Finally, if there have been enough enough time, Autotest should be
 enhanced so that the test scripts it generates become able to produce
 TAP-compatible output.


Plan and Roadmap
================

 Q: How will you and your mentor track your progress as you work on the
    project?
 A: Since technical discussions about design and implementation of new
    Automake features take place on the public lists <address@hidden>
    and/or <address@hidden>, most measurable progresses are
    publicly available there, and thus quite easy to judge.  Also, every
    new feature going in Automake must have proper testsuite coverage,
    so that we can be confident that the new code will really works.
    Finally, the project mentor has been the Automake maintainer since
    4/5 years now, so that he's surely able to judge how a partial
    implementation if faring, and what the chances are for it to be
    completed in time.

 Q: How the mid-term evaluation of your project will be made?
 A: I must have an implementation of Automake-generated TAP-consumer
    testsuite harness that works at least with non-obsolete versions of
    GNU make at least on GNU/Linux, Solaris 10 and FreeBSD 8.  This
    implementation must comprise test cases (maybe non exhaustive) and
    documentation (also  unpolished, but basically complete).

 Q: Remember to mention any periods during the summer when you won't
    actually be available to work on the project (though remember, the
    Summer of Code project is expected to be your main activity).
 A: I plan to be available for basically all the period of the project.
    If not the whole day, at least half a day, basically each day (either
    morning and early afternoon, or later afternoon and evening).  There
    might be *very few* short periods (one, max two days) when I'll be
    completely off-line and "away from keyboard" -- I'll let the mentor
    know about them in advance, obviously.


Communication
=============

  First, I'm already in contact with the mentor, due to my on-going
  involvement with the Automake project.  I've also been in contact
  with the backup mentor, due to my (small) contributions to Autoconf.

  In the Automake and Autoconf projects, discussions about design and
  documentation, and proposals and reviews of patches are public, taking
  place on the projects' pre-existing and well-established mailing lists.
  The mentor reads them regularly, so that posting patches an ideas there
  is a sure way to have them considered and reviewed by him (and possibly
  also by other mailing list readers, which might contribute with their
  own useful insights, questions, and experiences).

  Finally, my experiences have showed that real-time interaction with
  the mentor to work on a particular issue (either on-list or off-list)
  is not only possible, but also efficient and effective.


Qualification
=============

 I'm quite familiar with the policies of both the Automake and Autoconf
 projects (especially coding standards, patch review process, and
 portability concerns).  Moreover, the copyright assignment to the Free
 Software Foundation that allows me to contribute code to the Automake
 and Autoconf official repositories is already in place (and has been
 for more than a year and a half now).

 I've started contributing to the Automake project an year and a half
 ago.  During the last year, my involvement with the project has steadily
 increased: I've posted more patches and participated more actively to
 discussions on the mailing lists, I've been granted commit rights, and
 I've started helping with the bug tracker and related issues.  I plan
 to continue my involvement with Automake in the foreseeable future.

 I have access to some systems to test my work on: Debian GNU/Linux
 (with custom installation of older version of tools and compilers),
 FreeBSD 8.0, and Solaris 10.  This can help me in finding out early
 potential portability problems.

 I believe that powerful, easy-to-use testing frameworks are vital for
 the development of modern software.  Also, I find software testing an
 interesting, and sometimes even fascinating, topic.  So I can say I'm
 motivated enough to work in this area.

Tentative roadmap for GSoC project "automake - Interfacing with a test
protocol like TAP or subunit".


1. COLLECT INFORMATION: Before April 20
----------------------------------------

 Read carefully the TAP specifications available.  Skim the wiki at
 <http://testanything.org/>, and pick out useful pointers and information.
 
 Take a look at real-word usages of TAP (mostly the distribution of Perl
 and some major perl packages); familiarize as much as possible with the
 corner cases and the "rough edges", jot down some notes about things
 that are unclear might be deemed to cause potential problem.

 Finally, take a look at existing implementation (chiefly Test::Harness
 and c-tap-harness -- their licences are GPL-compatible, so that's OK
 from a legal point of view).


2. EXPERIMENTING: April 20 - May 15  (Before the official coding time)
-----------------------------------------------------------------------

 Write scripts (in shell and/or awk) and Makefiles that implement TAP
 consumers (it's OK to do so by tweaking Automake-generated Makefiles,
 and it's ok if they're not 100% functional).

 Try to make those examples as portable as possible, to spot potential
 portability issues early.  Test at least on GNU/Linux, FreeBSD 8 and
 Solaris 10, and their native make implementation(s).

 Exchange early ideas and doubts with the mentor, and, if appropriate,
 also with the official mailing lists.

 If there's still time, take a deeper look at Autotest documentation,
 implementation, and history, so that I will be able to (try to) extend
 it to produce TAP-compatible output (time permitting).

 This phase should help with the subsequent "real" implementation, by
 providing some ideas, and quite likely also code (incomplete of course,
 but probably good as a starting point and/or for "cut & paste"), and
 tentative testcases.


3. BASIC DESIGN: May 15 - June 5  (Official coding period starts May 23)
-------------------------------------------------------------------------

 We'll have to decide which is the best way to integrate the TAP
 consumer code into Automake -- will we use new primary, a new option,
 or maybe some new special syntax and/or variables?  This is something
 to be discussed publicly on the Automake mailing lists.

 Also, how much of the parallel-tests implementation and scaffolding
 can be reused?  What about parallelization and/or synchronization
 issues?  What about other issues that came up in the experimentation
 phase?  All these doubts should be worked out.
 
 If there's time, we might also start planning how Autotest can be
 extended to allow it to produce TAP output.  This is something to be
 discussed publicly on the Autoconf mailing lists.

 BTW, at this point, we'll probably need to go back and do some more
 experimenting (but more purpose-oriented than in the previous phase).
 And we could also write some early test cases and documentation.


4. WRITE BASIC TESTCASES AND DOCUMENTATION: June 5 - June 12
-------------------------------------------------------------

 Documentation about design and general concepts shouldn't be long
 nor verbose, but it's important to get it right and to make it as
 clear as possible -- which can be tricky, and take some time.  Also,
 condensing the most important use cases into tests is not immediate
 (although it shouldn't be very difficult in this phase).


5. CODING, PART 1: June 12 - July 6  (Until mid-term evaluation)
----------------------------------------------------------------

 Here I'll have to extend Automake to provide TAP-aware test harnesses.
 The resulting code must behave as to match the existing documentation
 and design.  It should also be able to generate Makefiles that work
 on at least GNU/Linux, FreeBSD and Solaris, with their native make
 implementation(s).  If this fails, I *must* at least be able to ensure
 that the generated Makefiles will work with all non-obsolete versions
 of GNU make on all those platforms.

 This is not a clear-cut phase.  During it, I'll have to be ready to
 extend and adjust the new documentation; adapt the existing testcases
 and add some more (probably many more); read and understand parts of
 the Automake codebase I'm not familiar with yet; and maybe doing more
 experimentation.  In the very worst case, the mentor and I might even
 be forced to revisit and modify our previous design.


5. CODING, PART 2: July 6 - July 23
------------------------------------

 If the support for TAP-aware test harnesses I wrote in the previous
 phase for Automake is complete and wholly functional (which include
 portability to at least BSD and Solaris make, but allows for some
 rough edges), in this phase I'll try to enhance Autotest to enable
 TAP-compatible output in its generated test scripts.  Otherwise,
 I'll have to work more on Automake TAP support.


6. DOCUMENTATION POLISHING: July 23 - July 28
----------------------------------------------

 Polish/complete documentation of all the implemented features.  Maybe
 add some examples, and if there's time contribute them to existing
 third-party tutorials, FAQs and/or wikis about the autotools.


7. STRESS AND PORTABILITY TESTING: July 28 - Until done
--------------------------------------------------------

 Test the code on other systems which Automake strives to support (e.g.,
 Cygwin, MSYS/MinGW, AIX, HP-UX, IRIX), and (try to) fix all the problems
 and incompatibilities that show up.  This might require some more help
 from the mentor, which has access to an impressive array of testing
 systems (all the major proprietary Unix implementations are among them).


8. FINAL CLEANUP: All the time that's left (if any)
----------------------------------------------------

 Note that this last pass is not strictly necessary, and might be omitted
 altogether in case we're out of time.

 Here we could add more tests about corner cases; polish documentation
 and/or code; and maybe even convert (part of) Automake's own testsuite
 to the use of the newly implemented TAP-based harness.