RE: Autmoake (test) editing

[Arthur Schwarz]

Editorial comments on TAP

================================================================

   
15.4.1 Introduction to TAP, pg. 112
   pg. 112 "... among different test harnesses ..."
      Automake has defined only one test harness, the Parallel Test Harness.
Since this is an Automake document mentioning of others makes no sense. The
sentence should be deleted or changed.
      
   pg. 112 "... will present the results on the console in the usual fashion
..."
      What is probably meant is that the TAP output will be presented. the
sentence otherwise doesn't make sense.
      
   pg. 112 "(see [Testsuite progress on console], page 103)"   
      There is no Testuite progress on console, and the hyperlink is to the
Parallel Test Harness. The Custom Driver, which is not a test harness, has a
Section 15.3.3.3 Testsuite progress output on pg. 111 and this is exclusive
to the custom driver. Perhaps the "usual fashion" needs some description.

   pg. 112 "(see [Basics of test metadata], page 105"
      There is no section titled "Basics of test metadata" and the hyperlink
is to the Parallel Test Harness, 15.2.3 Parallel Test Harness pg. 105. There
are various mentions to the .trs file and the contents of the .trs file.
References to the .trs file are on pgs. 105, 106, 107, 108, 110. A
description of the contents of a .trs file is given in 15.3.3.2 Log files
generation and test results recording, pg. 110 but this description is
defined for the custom driver and, one would presume, is exclusive to the
custom driver. There is no definition of the contents of a .trs file defined
for use by TAP (or anything other than the custom driver). What was the
intent of this sentence?
      
   pg. 112 "Apart from that, it will try to remain as much compatible as
possible with pre-existing and widespread utilities, such as the prove
utility, at least for the simpler usages."
      This is a description of the Automake generated test (something) and
its use by the user. If other products are involved then the intent of their
involvement should be defined and the expected input to these product
specified. My guess is that the sentence means that the .trs file data is
available to products external to the Automake generated test facilities for
analysis. I think that you probably mean to include the .log files also. And
that this data can be configured by the test case/test suite execution and
analysis software to conform to the data input requirement for these tools.
I have not idea what this has to do with Automake.
   
   pg. 112 "... various independent implementations in different languages
..."
      Which implementations are supported by Automake?
      
   pg. 112 "'Test::Harness::TAP'"
      This hyperlink no longer works.
http://testanything.org/tap-specification.html would probably work better.
      
   pg. 112 "... The most relevant real-world usages of TAP are obviously in
the testsuites of perl ..."
      This says that unless Automakee is generates code to test a perl
development, that TAP is not relevant. Is this really the intent? To
describe a facility which is not "most relevant" other than the restricted
environment of perl development?
      
15.4.2 Use TAP with the Automake test harness, pg. 112
   The only, and I mean only, test harness so far defined is the Parallel
Test Harness. Do you mean that TAP execution is concurrent?
   
   pg. 112 "... support for third-party test drivers ..."
      Is there a reference to integrating third-party test drivers? There is
a table of contents entry for third party Makefiles. The index has a
reference to integration with third-part packages, which seems to lead to
third-party Makefiles, and the text has references to various third-party
things, but I really don't see anything dealing with integrating third party
test drivers, unless you, perhaps, mean custom test drivers?
      
   pg. 112 "Apart from the options common to all the Automake test drivers
..."
      The reference, 15.3.3.1 Command-line arguments for test drivers, pg.
110, is to command-line interfaces to custom drivers. Are all Automake
drivers custom drivers? Makes sense but custom drivers aren't Automake
drivers. They are developer constructed drivers which Automake provides an
integrated interface for. there still is no definition of what a 'driver'
is. Can it be a script or a C/C++ program?
      
   pg. 112 "... tap-driver.sh supports the following options ..."
      No mechanism has been presented to tell how the user/developer inputs
options (however there is an example below, with no explanation). If it is
the developer, is it through autoconf/reconf options (clearly not allowed)?
if it is the user is it through 'make' options? And, there is no distinction
drawn here between the developer (an automake user) and the user (a Automake
generated make file user).
      
   pg. 113 "Here is an example  ..."
      You are using an example, with no description, to define how to
integrate TAP into Automake. Some explanation is required to understand the
impact and use of different constructs so that a developer can understand
this example. What is absolutely bad is to show
AC_REQUIRE_AUX_FILE([tap-driver.sh]) in configure.ac and TEST_LOG_DRIVER =
env AM_TAP_AWK='$(AWK)' $(SHELL) (top_srcdir)/build-aux/tap-driver.sh in
Makefile.am and expect someone using this manual for the first time to
understand what you are trying to do. This shouldn't be a classroom exercise
in exploration. What the user doesn't know the manual should teach. This is
a professional environment and your tools should be a professional document
explaining the use of product.
      
   pg. 113 "% env TEST_LOG_DRIVER_FLAGS='--comments --ignore-exit'
TESTS='foo.test baz.test' make -e check"
      The example provides a means for a user to interface with TAP. How
wonderful.

15.4.3 Incompatibilities with other TAP parsers and drivers, pg. 115
   pg. 1 "... pragma ... "
      The referenced TAP standard does not have pragma defined.
      
   pg. 115 "The --diagnostic-string ..."
      Grammatically incorrect.  Change "allows to modify" to "modifies".

15.4.4 Links and external resources on TAP, pg, 115
   pg. 115 "Test::Harness::TAP"
      Hyperlink not active.
      
   pg. 115 "The TAP wiki"
      Hyperlink no longer active.
      
      
   pg. 115 ""gentle introduction"
      Title changed to "Test::Tutorial - A tutorial about writing really
basic tests"