automake
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: TAP support in automake


From: Stefano Lattarini
Subject: Re: TAP support in automake
Date: Sun, 01 Apr 2012 00:35:44 +0200

On 03/31/2012 10:23 PM, Bob Friesenhahn wrote:
> On Sat, 31 Mar 2012, Stefano Lattarini wrote:
>>>
>> Well, automake being automake, it only depends on portable awk[1] ;-)
>> OK, portable "nawk" actually (as found by the autoconf builtin macro
>> AC_PROG_AWK), but I don't know of any non-museum machine lacking that.
> 
> Good.
> 
>> The new feature is already documented in the automake manual on the
>> 'master' branch; and this documentation should be release-quality
>> already.  But maybe I should make this new feature more visible before
>> the 1.12 release?  If yes, how do you suggest to do so?
> 
> Certainly mentioning and discussing it again on the 'automake' list 
> (happening now) is useful.
>
Let's continue the discussion then :-)

> The previous discussion I found was from almost a year ago before
> development of the feature was started.
> 
Communication failure on my part then :-(  The feature's development was
basically complete by the end of August 2011 (as mandated by the GSoC
deadlines).  And Automake has been happily using TAP in its own testsuite
since then!

>> As for the implementation quality, there's still an annoying glitch in
>> that you need to *manually* copy the automake-provided 'tap-driver.sh'
>> script yourself into your package[2] -- "automake --add-missing" won't
>> work -- and define some variables by hand; all of this is documented
>> in the manual, and should hopefully be fixed for automake 1.12.1.
> 
> This does not seem like a big problem.
> 
> Russ Allbery's C TAP Harness looks reasonable as well.  It also
> requires adding a single file (.c) to the source tree and a rule
> to build it as a test program.
>
Consider that the whole point of the TAP support in Automake is that the
testsuite harness present in Automake-generated makefiles (the one that
gets launched by "make check") will act as TAP consumer itself.  Thus
the "harness" part of Russ' package is likely of little or no use in
conjunction with the newer Automake test harness -- but of course you
can reuse the part of his project that provide a TAP *producer* for C
and/or Shell programs.

> The philosophy of TAP is rather different than Automake's existing
> test suite.
>
Yes, but not too radically IMHO; in fact, the TAP consumer implemented by
Automake reuses most of the code and interfaces of the 'parallel-tests'
test harness (that was an early design choice, and so far I have been very
happy with it).

> These differences may prove challenging when switching from Automake
> tests to TAP:
>
Note that you don't need any "flag day switch" for a conversion to TAP: "simple
tests" (whose outcome is purely based on their exit status) and TAP tests can
happily co-exists (as they currently do in the Automake's own testsuite).  This
too is an early, deliberate design choice.

>   * Every TAP test needs to somehow know its unique test number (so it
>     can print a result "ok 35").
>
Actually, not really:

  $ cat foo.sh
  #!/bin/sh
  echo 1..3
  echo 'ok'
  echo 'ok # SKIP'
  echo 'not ok # TODO'

  $ prove ./foo.sh
  ./foo.sh .. ok
  All tests successful.
  Files=1, Tests=3,  1 wallclock secs ( 0.04 usr +  0.00 sys =  0.04 CPU)
  Result: PASS

But as it's reasonable, if you provide a test number, that must be correct
(this should be undeniably recognized as a feature IMO); see:

  $ cat bar.sh
  #!/bin/sh
  echo 1..3
  echo 'ok 4'
  echo 'ok # SKIP'
  echo 'not ok # TODO'

  $ prove ./bar.sh
  ./bar.sh .. All 3 subtests passed
          (less 1 skipped subtest: 2 okay)
  Test Summary Report
  -------------------
  ./bar.sh (Wstat: 0 Tests: 3 Failed: 0)
    Parse errors: Tests out of sequence.  Found (4) but expected (1)
  Files=1, Tests=3,  0 wallclock secs ( 0.04 usr +  0.00 sys =  0.04 CPU)
  Result: FAIL

>     If the tests are all in the same
>     module then they can simply increment an integer but if they are
>     not then there is a problem (or they can always plan for one
>     test).
>
Apart from the fact that I'd advise against writing your TAP stream by
hand (you should use a small but higher-level module or library or set
of functions to do so), the situation you describe shouldn't be a problem
in any case: Automake-produced Makefiles fire off a *new* TAP-parsing
process for each TAP-based test script in $(TESTS) [1], so that you would
only need to worry about the ordering/numbering of the test cases in the
script you are modifying.

 [1] This is also extremely important in order to preserve the ability
     to run tests in parallel -- one of the major selling points of the
     Automake testsuite harness.

>   * TAP tests need to know if they are expected to pass or fail.
> 
> A major feature of how GraphicsMagick is using the Automake tests
> framework is that it determines which tests are XFAIL based on what
> features were configured into the software.  The list of tests which
> are expected to fail is produced based on Automake conditionals.
> It is useful for the user to see tests XFAIL as a reminder that the
> configuration may be missing something they wanted.  How is this best
> handled for TAP tests?
>
That is highly dependent on the exact details of how your testsuite is
organized; if you have time to present more precise/targeted questions
in this respect, I'd be happy to help.

(And I see now that Russ has already given some good advice, the I feel
to endorse).

> Bob

Regards,
  Stefano



reply via email to

[Prev in Thread] Current Thread [Next in Thread]