Re: [GSoC Proposal] automake - Interfacing with a test protocol like TAP or subunit

[Robert Collins]

On Sun, Mar 20, 2011 at 10:01 PM, Stefano Lattarini
<address@hidden> wrote:
> Hello Robert, and thanks for the feedback.
>
> On Sunday 20 March 2011, Robert Collins wrote:
>> On Sat, Mar 19, 2011 at 1:03 PM, Stefano Lattarini
>> <address@hidden> wrote:
>> > ABSTRACT:
>> >
>> >  The Test Anything Protocol (TAP) is a simple text-based protocol
>> >  that allows communication between test scripts and a test harness.
>> ...
>> > Now, in all honesty, I must say that I've chosen TAP not only for its
>> > objective qualities and merits, but also because I have some previous
>> > knowledge of it (which allowed me to present a more concrete proposal
>> > and a meaningful roadmap) and I personally like it (which will probably
>> > be a powerful motivator to overcome the unavoidable hurdles I'll
>> > encounter down the road).
>> >
>> > Still, there might be very valid competing alternatives to TAP out there,
>> > which I might not know about, but that in the long run would offer
>> > Automake more advantages and interoperabilty, thus outweighting the two
>> > "personal" advantages of TAP I've reported above.  So, if anyone who's
>> > reading this has    proposals about viable alternatives to TAP, please
>> > speak up -- your contribution is appreciated!
>>
>> TAP is an extremely simple protocol, and the extensions to it to
>> support things like not needing to maintain the count of tests,
>> additional debug data and so on are pretty rudimentary. subunit, which
>> I've mentioned before was written after TAP, to solve similar problems
>> and address the issues in TAP itself.
>>
>> While the users of a protocol aren't really an indicator of the
>> protocols worth, projects as large as samba are using subunit.
>>
>> Unlike TAP subunit supports attachments (binary and text) to tests,
>> test naming, tagging, timestamping (permitting robust timing data even
>> in parallelised or distributed testing).
>>
> Is there some document that describes the subunit features in depth *and*
> with examples?  I mean something similar to what the following is for TAP:
>  <http://search.cpan.org/~petdance/Test-Harness/lib/Test/Harness/TAP.pod>
> That would be very useful to do a proper comparison, and weight advantages
> and disadvatnages.

http://bazaar.launchpad.net/~subunit/subunit/trunk/view/head:/README#L105 and
http://bazaar.launchpad.net/~subunit/subunit/trunk/view/head:/README#L134
may cover what you're asking.

>> Integrating with TAP is basically uninteresting to anyone working in a
>> high level language: Python, Ruby,Java, C++ etc.
>>
> How so? (honest question)

Python tests are most often written using one of:
 - unittest (built in)
 - doctest (built in, horrible ;))
 - py.test (external, quite popular)
 - nose/trial/zope.testing/any of a number of other third party
extensions that use the basic unittest object model: that an xUnit
model,  and TAP does a very poor job of interfaces with xUnit runners.
 - there are some other custom test runners around, including the TAP
one, but they (IME) have nearly-no uptake.

Java has the same xUnit test runner model, and it dominates the
testing frameworks there - massively and comprehensively. All the
domain specific, literate testing, BDD testing frameworks I've seen
are built to be compatible with junit.

Ruby seems to be dominated by rspec and cucumber, both of which are
entire runners in their own right.

C++ has cppunit and a recent addition, the google test runner, as its
dominate test environments. CPPUnit is another xUnit framework, as is
googletest.

There are four primary targets for test run integration:
 - local IDE : tests can drop straight into a debugger : things like
TAP and subunit generally just get in the way.
 - local via a subprocess : subunit/TAP style filtering and chain can
be useful to get effective output
 - distributor / end user builds : report success/failure and provide
a diagnostic to return to the developer
 - automated test runs - Jenkins/Buildbot/parallel test farms :
provide effective introspection of all test run, and as much
environmental data as possible to allow effective reproduction and
correction

The overly simple model of TAP does not fit well with the xUnit style
of tests - and its this friction that to me explains why TAP really
seems isolated to the Perl community (where it is extremely popular).

-Rob