Re: [GSoC Proposal] automake - Interfacing with a test protocol like TAP or subunit

[Robert Collins]

On Sun, Mar 20, 2011 at 8:53 PM, Ralf Wildenhues <address@hidden> wrote:
> Hi Robert,
>
> thanks for the feedback.  I have a couple of questions:
>
> * Robert Collins wrote on Sun, Mar 20, 2011 at 05:10:16AM CET:
>> TAP is an extremely simple protocol, and the extensions to it to
>> support things like not needing to maintain the count of tests,
>> additional debug data and so on are pretty rudimentary. subunit, which
>> I've mentioned before was written after TAP, to solve similar problems
>> and address the issues in TAP itself.
>
> Are TAP and subunit compatible on their common subset?  If not, why not?

You can convert TAP to subunit, and you can convert the things TAP can
represent from subunit to TAP. subunit's core is more structured than
TAP, so the two protocols don't pun as each other at all.

>> Unlike TAP subunit supports attachments (binary and text) to tests,
>
> Handling of binary data may end up being quite tricky within a
> restricted Posix environment with only a few tools available.
> What if a consumer cannot handle them?  Is there possibility
> for graceful fallback?

I didn't think posix had 7 bit limits? Anyhow, there is a baseline
profile which assumes just a single description of the error in a test
- it uses \r] to delimit a traceback. Alternatively, a C parser - on
my 'sometime' TODO list - will probably clock in small enough to
bundle for projects very low in the dependency stack.

I think a reasonable approach would be:
 - have a tiny minimal 'list test and report' consumer which can be
bundled for the bootstrap closure of projects.
 - detect (via autoconf) the presence of existing subunit consumers
and preferentially use those.

However, even this may not be needed- most bring-up situations involve
cross compiling these days, so just depending on the presence of an
executable subunit-stats (for instance) may be sufficient : cross
compile a set of C based subunit consumers, done.

>> test naming, tagging, timestamping (permitting robust timing data even
>> in parallelised or distributed testing).
>>
>> Integrating with TAP is basically uninteresting to anyone working in a
>> high level language: Python, Ruby,Java, C++ etc.
>
> Well, Perl is a high-level language too, I'd say.  ;-)

It is and it isn't - it has a very-close-to-the-metal feel to it. perl
6 may change this more dramatically :).

-Rob