RE: What is a 'program' in Section 15.3.3.1?

[Bob Friesenhahn]

On Wed, 27 May 2015, Arthur Schwarz wrote:
> TAP is supposed to be a custom test driver. Is the interface and manner of
> calling different from other custom drivers, and the API and other comments
> describing custom drivers in the manual?

I think that the method of calling is similar to other type of tests 
except that the list of tests 'TESTS' is a list of scripts which 
produce TAP output strings.

> > It depends on subordinate
> > scripts printing
>
> I'm very confused here. It looks to me like tap-driver.sh is a standalone
> script and doesn't need any help scripts. The input data is processed in awk
> and all the needed functions are defined in this context.

If the test program was a C program so it was a binary and printed 
'ok' and 'not ok' outputs, and is able to find any input files without 
assistance, then no extra shim script should be required.

In my case I was replacing perhaps 1000 individual classic Automake 
test scripts and wanted to replace them with far fewer TAP scripts.

> I am confused. Using Automake the Developer can generate a reference to a
> class of test cases defined in the TESTS variable. Each one of the scripts
> is required to output the results of one of more tests that they run in
> quasi-TAP format. The TAP script, tap-driver.sh, takes the output and
> generates a trs file. Included with the tap-driver.sh there is a means to
> generate XFAIL and XPASS, however this seems to be global to all subtests
> executed in a TESTS defined test. Each ok or not ok returned will be
> translated (as required) to XPASS or XFAIL depending on
> --expect-failure=yes.

A TAP test program/script may contain many tests and it may output 
multiple test lines.  The script indicates how many tests it plans to 
run by first printing a line like

1..532

to indicate that it plans to run 532 tests.  Each test gets is own 
PASS, FAIL, XFAIL indication.

> As an example:
> TEST_EXTENSIONS =.abc
> ABC_LOG_DRIVER = tap-driver.sh
> TESTS = test1.abc test2.abc
>
> Means that tap-driver.sh is called twice, once for test1.abc, once for
> test2.abc. Each of these tests can return one or more of ok, not ok, skip,
> bail, or a YAML-like text string defined by --diagnostic-string and
> --comments.

Yes, but tap-driver.sh interprets the output while the test 
program/scripts runs and it can handle an arbitary number of 
tests/results in the same test script.

> Anyhow, thanks for the heads up, I will be looking at your files. And, as
> always, I am confused.

I am not surprised. :-)

Bob
--
Bob Friesenhahn
address@hidden, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/