autoconf-patches
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: autotest annoyances


From: Eric Blake
Subject: Re: autotest annoyances
Date: Sat, 08 Apr 2006 14:57:11 +0000

Hi Ralf,
> > While testing my previous patch for const/volatile vs. -Werror, I 
> > encountered
> > two autotest anomalies.  I first enhanced the testsuite to detect both
> > problems, as well as my earlier patch this week for when removing
> > the test group directory fails.
> 
> It's not completely obvious to me that your tests also detect the
> earlier failure.  Is that because they happen to use `./run' at all?
> If so, then that is ok.

My earlier patch (AS_MKDIR_P instead of mkdir, because rm can fail)
is tested by the fact that I call ./run from within the directory (effectively
locking it on cygwin, as a good way to provoke a failed rm).  True,
on platforms like Linux where another process can rmdir the working
directory of the first, that patch remains untested, since there is not
an easy way to provoke the rm failure.

But you are also right that my first test case has a problem (I realized
it last night, and would have sent a fix even if you had not caught it).
The problem is that it mixes two failure modes (run not being recreated,
and run being created incorrectly), but is not explicit enough to tell the
two modes apart.  So it was accidentally passing since ./micro-suite
complained about an unknown option with exit status 1, instead of
rerunning the test group and getting an unexpected failure with exit
status 1; compared to the pre-test failure where run didn't exist so
rerunning failed with status 127.  I am working on a resubmission,
and it will make sure that the pass is correct, not accidental.

To do so, I would like group consensus on the following proposal:
testsuite currently has 2 exit statuses - 0 for all requested tests ran
as expected (even if no tests were requested, such as when --help
is given); 1 for any unexpected results, or for argument parsing
problems.  I would like to change this to 3 statuses:
0 for all requested tests ran as expected (pass, skip, xfail)
1 for at least one unexpected test result (fail, xpass)
2 for argument parsing or other serious problems

But this introduces another wrinkle - failure of AS_MKDIR_P
would qualify for an exit status of 2, but it is hard-wired to use
AS_ERROR with its default exit status of 1.  So I would also
need to add AS_ERROR_EXIT_STATUS - when defined, it
replaces 1 as the default exit status used by AS_ERROR.
Then autotest would just ensure that AS_ERROR_EXIT_STATUS
is defined to 2 for uses of AS_ERROR not related to running
test groups.

Obviously, this would also need more tests in autotest.at.
And I think the first test can be tightened to test only
missing ./run, instead of getting mixed up with invalid ./run,
 even if my proposal is too invasive before the alpha release.

> 
> > The first bug was that when using the -d option to testsuite (which is done 
> > by
> > all the testsuite.dir/###/run scripts), the run script was only regenerated
> > if the test failed (it was missing after a success or skip).
> 
> Hmm.  Both `./testsuite --help' and the documentation state explicitly
> that `-d' prevents creation of the `run' script in the first place.

But that documentation made it sound like testsuite.dir/*/{run,testsuite.log}
would be preserved, whereas what actually happens is that we do
rm -rf testsuite.dir to remove any cruft from the previous run.  To make
that work, we would first have to check if run or testsuite.log exist and
move them to a temporary location before removing the testgroup
directory, then move them back when the test has rerun.

> I see that this may not be the best idea to keep that, since I don't
> know why using `./run' once should not be idempotent, i.e., result in
> the same state after rerunning the test.  But anyway the documentation
> needs to be fixed; but first we should come up with a sane set of
> semantics.

So my resubmittal will have to touch up that documentation as well.
I think that the best semantics is that running the debugging script
regenerates the entire directory, including a fresh rebuild of ./run,
and with a new timestamp for testsuite.log; but that since the
re-run uses the same command line options, the same results
should happen.

> 
> > The second bug is that environment variables passed as options to 
> > ./testsuite
> > are copied directly into the run scripts, but without regards for shell
> > metacharacters.  That means that an environment variable that contains
> > spaces, such as "./run CFLAGS='-std=c89 -Werror'", will regenerate a broken
> > run script that tries to invoke 'testsuite CFLAGS=-std=c89 -Werror'.  
> > However,
> > fixing this quickly and safely is beyond my abilities, with a pending alpha
> > release, so the test is just xfailed for now.
> 
> This could be fixed similarly to how lib/autoconf/general.m4 quotes
> the contents of ac_configure_args selectively for config.status.
> A bit tricky, but it's possible.  (It's not safe against all kinds
> of uses of backslashes; that would require searching for an echo
> replacement like Libtool does.)

I'll start looking into this, but for now I will leave it as an xfail and
focus instead on the issues raised by the first bug.

-- 
Eric Blake




reply via email to

[Prev in Thread] Current Thread [Next in Thread]