automake
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

RE: Question on --test-name=NAME


From: Arthur Schwarz
Subject: RE: Question on --test-name=NAME
Date: Sun, 26 Apr 2015 10:39:53 -0700

> 
> 
> -----Original Message-----
> From: Gavin Smith [mailto:address@hidden 
> Sent: Sunday, April 26, 2015 9:03 AM
> To: Arthur Schwarz
> Cc: Automake Mailing List
> Subject: Re: Question on --test-name=NAME
> 
> On 26 April 2015 at 16:30, Arthur Schwarz <address@hidden> wrote:
> >
> >
> > 15.3.3.1 Command-line arguments for test drivers
> >
> >    --test-name=NAME
> >
> >        The name of the test, with VPATH prefix (if any) removed. This
can
> > have a suffix and a directory component (as in e.g., sub/foo.test), and
is
> > mostly meant to be used in console reports about testsuite advancements
and
> > results (see Testsuite progress output).
> >
> >
> > If test-name is _mostly meant to be used in console reports_ why is
there
> > any concern for a path? One can use any string in a report, paths are
> > unnecessary, and you haven't specified any other use.
> >
> > So let me guess what you might be trying to say. From 15.3.1 Overview of
> > Custom Test Drivers Support it appears that the test handler
(responsible
> > for test suite summaries and I suspect for executing test drivers)
executes
> > the _custom test driver_ which executes a test program, which must be a
> > _test script_. The test script usage is directly contradicted in 15.3.2
> > Declaring Custom Test Drivers where _ext_ may or may not be a script
suffix,
> > but whose counting.
> >
> > The _custom test driver_ is responsible for interpreting command line
> > arguments. One assumption is that when this argument is received the
_custom
> > test driver_ executes the NAME given in the command line argument. Of
> > course, if this were true then it is possible to input a NAME which is
not
> > one of the known tests given by the developer. I don't know that this
makes
> > much sense, to execute a random script not defined by the developer.
> 
> After the list of options it says:
> 
> "The first non-option argument passed to the test driver is the program
> to be run, and all the following ones are command-line options and
> arguments for this program."
> 
> By "program" it means test script.
> 
> Maybe this could be clearer with a synopsis, like "DRIVER
> DRIVER-OPTIONS -- TEST [TEST-OPTIONS]". It could be more explicit that
> the test driver has to run the program it is given: maybe
> 
> "The first non-option argument passed to the test driver is the test
> script for it to run".
>

In Apologia: If I sound (come across as) sarcastic, acerbic and short its
because I'm frustrated. I've been at this since February and have gotten to
the point where I have to parse each sentence to determine what it means.
The document does not say what it means nor does it mean what it says. Your
document quote at the end is an example (see
http://lists.gnu.org/archive/html/automake/2015-04/msg00022.html thread).

So now, grab a whiskey, light a candle, find a chair and I'll begin.

In the beginning there was a test handler calling test drivers calling test
cases. Now there are several test handlers, each one identified by automake
in a different fashion. Only one test handler is active in any given
generated script.

The custom test driver is a test driver. It can be a script or a program
(see the description of TEST_EXTENSIONS and their relation to LOG_DRIVERS).

The custom test driver can call a test case. A test case can be a script or
an executable program (see the descriptions of TEST_EXTENSIONS and
LOG_DRIVERS and think on a windows *.exe test case being executed).

So at this point we have discovered that the manual is incorrect is
specifying that we must have a test script.

Now for the case you mentioned in identifying a program, which is really a
test case renamed in the manual for mystifying and unknowable reasons. Well,
there is no mechanism for "The first non-option argument passed to the test
driver is the program to be run, and all the following ones are command-line
options and arguments for this program." Let's go through some of options.

Suppose command line arguments are input on the command line (seems
reasonable huh). Note that none of the command line inputs are identified or
described in the make help file (make --help) or the make man pages (man
make). I haven't looked at info make but, things being what they are, their
probably not there either. So, since the manual doesn't describe how or what
to do I guess we get to make it all up.

In http://lists.gnu.org/archive/html/automake/2015-04/msg00022.html the
question was asked as to how a program (test script) is to be entered. Let's
see.

Option 1: make program check
Option 2: make program
Option 3: make --test-name=NAME program check
Option 4: make --test-name=NAME check
Option 5: make program --test-name=NAME check
Option 6: you can't do it

Pick one or invent your own! That's how the command line options must be
input. I can't think of another way.

Now, once we input the command line options (and programs/test case) we pass
it to the driver. Well, now we have something we can determine.

testDriver [options] program

That's what it (seems to) says and that's what it is. Only I can't figure
how to get to this point.

Remember that ext_LOG_DRIVER associates a driver (whatever that is) with a
class of things to be driven. We are not dealing with scripts. We are
dealing with everything, see 15.2.3 Parallel Test Harness parallel test
example, which is really a custom driver example. And note that an example
of custom test driver usage does not appear with the description of custom
test drivers.

Oh, about the example. The automake variable can't be used for anything but
scripts, see 15 Support for test suites "The other (and most used) form is
based on the use of generic test scripts, and its activation is triggered by
the definition of the special TESTS variable." This quite clearly says that
anytime that a TESTS variable is seen then generic scripts are used. But the
example has "TESTS = foo.pl bar.py baz" which are not generic scripts. It
must mean that the test driver is a generic script, but oh wait. The test
drivers are for PERL and PYTHON and scripts. And (last time I looked) PERL
and PYTHON were not generic scripts.

One more little teenie-weenie thingie.

Test handler types are recognized by all of:
1: Nothing. If there is nothing then the test handler executes bash on the
test cases. 
2: LOG anything. The test handler is for custom test drivers. And scripts
handlers must be provided.
3: AUTOMAKE_OPTIONS = serial-tests: The test handler is a serial test
handler (oh, by the way, the option is not in alphabetic order).
4: AUTOMAKE_OPTIONS = parallel-tests or missing, then the test handler is a
parallel test handler.
5: AUTOMAKE_OPTIONS = dejagnu: The test handler handles a dejagnu test
drivers?
6: TEST_LOG_DRIVER = env AM_TAP: The test handler is for TAP something.

And oh, I'm frustrated. I'm sorry if the verbiage is sarcastic. I hope you
will see how contradictory the manual is; how it lack precision and how it
appears as an ad hoc product.

art  




reply via email to

[Prev in Thread] Current Thread [Next in Thread]