Re: [GSoC Proposal] automake - Interfacing with a test protocol like TAP or subunit

[Stefano Lattarini]

On Sunday 20 March 2011, Robert Collins wrote:
> On Sun, Mar 20, 2011 at 10:01 PM, Stefano Lattarini
> <address@hidden> wrote:
> > Hello Robert, and thanks for the feedback.
> >
> > On Sunday 20 March 2011, Robert Collins wrote:
> >> On Sat, Mar 19, 2011 at 1:03 PM, Stefano Lattarini
> >> <address@hidden> wrote:
> >> > ABSTRACT:
> >> >
> >> >  The Test Anything Protocol (TAP) is a simple text-based protocol
> >> >  that allows communication between test scripts and a test harness.
> >> ...
> >> > Now, in all honesty, I must say that I've chosen TAP not only for its
> >> > objective qualities and merits, but also because I have some previous
> >> > knowledge of it (which allowed me to present a more concrete proposal
> >> > and a meaningful roadmap) and I personally like it (which will probably
> >> > be a powerful motivator to overcome the unavoidable hurdles I'll
> >> > encounter down the road).
> >> >
> >> > Still, there might be very valid competing alternatives to TAP out there,
> >> > which I might not know about, but that in the long run would offer
> >> > Automake more advantages and interoperabilty, thus outweighting the two
> >> > "personal" advantages of TAP I've reported above.  So, if anyone who's
> >> > reading this has    proposals about viable alternatives to TAP, please
> >> > speak up -- your contribution is appreciated!
> >>
> >> TAP is an extremely simple protocol, and the extensions to it to
> >> support things like not needing to maintain the count of tests,
> >> additional debug data and so on are pretty rudimentary.
>
[About this I forgot to say:]
True, but sometimes "worse is better": TAP is (mostly?) line-based, and
POSIX/Unix tools are design to operate on a line-by-line bases, so its
clear what the path of the least resistance is here (BTW, I admit that,
with some twists and hacks, you can have many Unix utilities operate
in a more "multiline aware" fashion, but that usually requires subtantial
efforts, and produces code that is complex and hard to understand and
maintain).

> >> subunit, which
> >> I've mentioned before was written after TAP, to solve similar problems
> >> and address the issues in TAP itself.
> >>
> >> While the users of a protocol aren't really an indicator of the
> >> protocols worth, projects as large as samba are using subunit.
> >>
> >> Unlike TAP subunit supports attachments (binary and text) to tests,
> >> test naming, tagging, timestamping (permitting robust timing data even
> >> in parallelised or distributed testing).
> >>
> > Is there some document that describes the subunit features in depth *and*
> > with examples?  I mean something similar to what the following is for TAP:
> >  <http://search.cpan.org/~petdance/Test-Harness/lib/Test/Harness/TAP.pod>
> > That would be very useful to do a proper comparison, and weight advantages
> > and disadvatnages.
> 
> http://bazaar.launchpad.net/~subunit/subunit/trunk/view/head:/README#L105 and
> http://bazaar.launchpad.net/~subunit/subunit/trunk/view/head:/README#L134
> may cover what you're asking.
>
Hmm... I was hoping for something more "meaty", especially considering that
SubUnit is more complex and powerful than TAP (BTW, I had already taken a
look at the README after I cloned the subunit repository; sorry for not
stating that explicitly).

> >> Integrating with TAP is basically uninteresting to anyone working in a
> >> high level language: Python, Ruby,Java, C++ etc.
> >>
> > How so? (honest question)
> 
> Python tests are most often written using one of:
>  - unittest (built in)
>  - doctest (built in, horrible ;))
>
Just curious: why horrible?  I've used it a few times, and I liked it
very much (but I've used it only from the corresponding 'py.test' plugin,
so my judgment might be skewed by this).

>  - py.test (external, quite popular)
>  - nose/trial/zope.testing/any of a number of other third party
>    extensions that use the basic unittest object model: that an xUnit
>    model, and TAP does a very poor job of interfaces with xUnit runners.
>
Is this due to intrinsic/architectural reasons, or because nobody has
written a "TAP plugin" for them yet?  IIUC, 'nose' ans 'py.test' are
designed to be "easily" extensible through plugins (I don't know 'trial'
and 'zope.testing', and plain unittest is not worth using anyway IMO --
python is not Java!).

>  - there are some other custom test runners around, including the TAP
>    one, but they (IME) have nearly-no uptake.
>
I agree that from a python user POV, the best approach is to use 'nose'
or 'py.test' (or comparable testing packages); it should be these
packages' task to output TAP (or SubUnit, or XML) output, if the user
requests so.

> Java has the same xUnit test runner model, and it dominates the
> testing frameworks there - massively and comprehensively.
>
[Java support in Automake is very limited anyway, so I wouldn't bother
 about that ATM.]
> All the domain specific, literate testing, BDD testing frameworks
> I've seen are built to be compatible with junit.
> 
> Ruby seems to be dominated by rspec and cucumber, both of which are
> entire runners in their own right.
> 
> C++ has cppunit and a recent addition, the google test runner, as its
> dominate test environments. CPPUnit is another xUnit framework, as is
> googletest.
>
But does any of these frameworks produce SubUnit output at the moment?
If not, SubUnit and TAP are in a tie in this respect...

Also, the great majority of Automake-based projects are C based, and use
C or the shell in the testsuite; since there are TAP-producing libraries
for those two languages -- see for example:
  <http://www.eyrie.org/~eagle/software/c-tap-harness/>
all those projects might benefit from starting to use TAP.  This will be
even more true if Autotest is enhanced to start producing TAP output.

> There are four primary targets for test run integration:
>  - local IDE : tests can drop straight into a debugger : things like
>    TAP and subunit generally just get in the way.
>  - local via a subprocess : subunit/TAP style filtering and chain can
>    be useful to get effective output
>  - distributor / end user builds : report success/failure and provide
>    a diagnostic to return to the developer
>  - automated test runs - Jenkins/Buildbot/parallel test farms :
>    provide effective introspection of all test run, and as much
>    environmental data as possible to allow effective reproduction and
>    correction
> 
> The overly simple model of TAP does not fit well with the xUnit style
> of tests - and its this friction that to me explains why TAP really
> seems isolated to the Perl community (where it is extremely popular).
>
> -Rob
> 

Regards,
  Stefano