automake
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: TAP support in automake


From: Stefano Lattarini
Subject: Re: TAP support in automake
Date: Sun, 01 Apr 2012 09:35:57 +0200

On 04/01/2012 02:50 AM, Bob Friesenhahn wrote:
> On Sun, 1 Apr 2012, Stefano Lattarini wrote:
> 
>> On 03/31/2012 11:47 PM, Bob Friesenhahn wrote:
>>>
>>> how does one re-execute just one test in order to see the details
>>> of how it failed?
>>>
>> With Automake, one shouldn't -- he should write his test to be verbose
>> enough so that a perusing of the logs is sufficient to reveal the reason
>> of the failure.
> 
> Ok, now I hear that there are still log files similar to parallel-tests
> (presumably using the identical facility).
>
Yes.  And also (newly-introduced) "trs" (Test ReSults) files that store
some metadata about the tests' runs (note however that the format of
these files is still highly exeprimental and subject to change).

> It does not seem that Russ's C TAP Harness offers this quite wonderful
> feature since the instructions advise to re-run just the one test to
> see more verbose output (similar to what I have experienced from perl).
> 
>> You mean running only a given test case in a given TAP test script, or all
>> the test cases in a given TAP test script?  Currently, there is no 
>> transparent
>> way way to do the former (which is hardly surprising), but for the latter,
>> you would use the same interface you use for "simple" tests:
>>
>>  make check TESTS=granular.tap
>>
>> or even:
>>
>>  make check TEST_LOGS=granular.log
> 
> So Automake TAP still supports the same parallel-tests facility but
> treats files with the ".tap" extension as being TAP test scripts?
>
Not by default; you must declare which extensions should use the TAP
driver, with the definition of a proper "<ext>_LOG_DRIVER" variable.

For example, if your testsuite contains "simple tests" written as shell
script (and having a '.sh' extension) and TAP tests written as perl
scripts (and having a '.pl' extension), your Makefile.am might look
like this:

  # Tell which program should run the .sh scripts.
  SH_LOG_COMPILER = $(SHELL) -ex
  # Tell which program should run the .pl scripts
  PL_LOG_COMPILER = $(PERL) -MTAP::More -w
  # Tell which program should interpret the outcomes for the .pl scripts
  PL_LOG_DRIVER = env AM_TAP_AWK='$(AWK)' $(SHELL) \
                  $(top_srcdir)/build-aux/tap-driver.sh

In the next Automake version, I hope to be able to refine the above
to something like this:

  SH_LOG_COMPILER = $(SHELL) -ex
  PL_LOG_COMPILER = $(PERL) -MTAP::More -w
  PL_LOG_PROTOCOL = tap

Of course, the use of "<ext>_LOG_DRIVER" variables will still be
supported, to allow developers to define their own drivers if they
want to (e.g., if they want to support a protocol like SubUnit that
is not supported by default by Automake).

> That sounds like a useful approach.
>
I'm glad to hear this.

> An issue with using the TAP facility to reduce the number of tests
> (to fix command line length issues) is that it reduces the amount
> of available test parallelism.
>
That's a risk if TAP is over-used or abused; it's up to the developer
to strike a balance.  You can take a look at the TAP usage in the
automake's own testsuite for few examples.

Regards,
  Stefano



reply via email to

[Prev in Thread] Current Thread [Next in Thread]