Re: Autotest: loops of tests

[Ralf Wildenhues]

This is continuing an old old thread from the autoconf list:
http://lists.gnu.org/archive/html/autoconf/2005-03/msg00117.html
I moved it to autoconf-patches now.

* Noah Misch wrote on Thu, Mar 24, 2005 at 03:47:45AM CET:
> On Tue, Mar 22, 2005 at 10:05:04AM +0100, Ralf Wildenhues wrote:
> > 
> > I'm not too happy with the replication they cause in `testsuite' -- one
> > of my ideas was to test on the order of 2^8 link flag combinations
> > eventually, and I would like to keep the blow-up as small as possible.
> 
> I understand.  Current Autotest does not provide a clean way to avoid that
> bloat.  That is a problem worth addressing; the damage to non-Autoconf users 
> of
> Autotest is particularly bad since you distribute `testsuite' itself.

I hacked something up.  Rather ugly for Autoconf standards, but it
does exactly what I think I need: it avoids all bloat.  If you can
make something nice out of it, the better.  I would really like to
see functionality comparable to this in autotest.

Currently, AT_CHECK(cmd, ignore, , , fail_commands)
is somewhat superfluous, as fail_commands will never get executed
because we ignore the return value.  The patch below changes that.
I regard this change in semantics as ok because of the nonsensical
behavior this combination had before, IMVHO.

It passes the test suite, and even makes `tests/testsuite' 33 bytes
smaller, by using a bit more m4 compile time.  :)

Do you like it?  It allows me to do this (as an example, a failure
is generated when $i is nonempty):

### snip loop.at ###
dnl autom4te -l autotest loop.at > loop; chmod +x loop; ./loop
m4_define([AT_PACKAGE_STRING],[loop])
m4_define([AT_PACKAGE_BUGREPORT],[devnull])
AT_INIT
AT_SETUP([loop])
AT_CAPTURE_FILE([failures])
for i in '' one '' two '' three; do
  dnl Stylistic issue: we overwrite `failures' here to avoid quadratic
  dnl growth of `testsuite.log' in the number of failures.
  dnl This is because many tests might fail for the same reason.
  AT_CHECK([test -z "$i"], ignore,ignore,ignore, [echo "$i" > failures])
done

dnl next command generates the real failure:
AT_CHECK([test -s failures],1,ignore,ignore)
AT_CLEANUP
### snip ###


The design decision whether or not to capture files in the case in
question is not clear to me -- I'd probably prefer yes, like I've
done now.  That way allows more flexibility for the commands.

Cheers,
Ralf

        * lib/autotest/general.m4 (AT_CHECK, _AT_CHECK):
        If RUN-IF-FAIL is given, execute it upon nonzero status, even
        if STATUS is set to `ignore'.  Capture files then, too.
        * doc/autoconf.texi (Writing testsuite.at): Updated.

Index: doc/autoconf.texi
===================================================================
RCS file: /cvsroot/autoconf/autoconf/doc/autoconf.texi,v
retrieving revision 1.925
diff -u -r1.925 autoconf.texi
--- doc/autoconf.texi   6 Jul 2005 21:39:31 -0000       1.925
+++ doc/autoconf.texi   14 Aug 2005 14:06:36 -0000
@@ -16168,8 +16180,10 @@
 The @var{commands} @emph{must not} redirect the standard output, nor the
 standard error.
 
-If @var{status}, or @var{stdout}, or @var{stderr} is @samp{ignore}, then
-the corresponding value is not checked.
+If @var{stdout} or @var{stderr} is @samp{ignore}, then the corresponding value
+is not checked.  If @var{status} is @samp{ignore}, a non-zero return value
+does not cause the test group to fail, but nevertheless causes
address@hidden to be executed, and files to be captured.
 
 The special value @samp{expout} for @var{stdout} means the expected
 output of the @var{commands} is the content of the file @file{expout}.
Index: lib/autotest/general.m4
===================================================================
RCS file: /cvsroot/autoconf/autoconf/lib/autotest/general.m4,v
retrieving revision 1.189
diff -u -r1.189 general.m4
--- lib/autotest/general.m4     27 Jul 2005 14:20:32 -0000      1.189
+++ lib/autotest/general.m4     14 Aug 2005 14:06:36 -0000
@@ -1207,7 +1207,9 @@
 # STDERR contents.  Shell metacharacters in STDOUT and STDERR are
 # _not_ processed by the shell, but are treated as string literals.
 #
-# STATUS, STDOUT, and STDERR are not checked if equal to `ignore'.
+# STDOUT and STDERR are not checked if equal to `ignore'.
+# If STATUS is equal to `ignore', non-zero return value does not cause
+# the test group to fail, but does RUN-IF-FAIL to be executed.
 #
 # If STDOUT is `expout', then stdout is compared to the content of the file
 # `expout'.  Likewise for STDERR and `experr'.
@@ -1437,7 +1439,7 @@
     [   77) echo 77 > "$at_status_file"; exit 77;;
 ])dnl
 m4_if([$2], [ignore],
-    [   *);;],
+    [   0);; *) at_failed=:;;],
     [   m4_default([$2], [0])) ;;
    *) echo "$at_srcdir/AT_LINE: exit code was $at_status, expected 
m4_default([$2], [0])"
       at_failed=:;;])
@@ -1446,7 +1448,8 @@
   m4_ifdef([AT_capture_files],
     [for file in AT_capture_files
      do echo "$file:"; sed 's/^/> /' "$file"; done])
-  echo 1 > "$at_status_file"
-  exit 1], [$6])
+  m4_if([$2], [ignore], [at_failed=false],
+  [echo 1 > "$at_status_file"
+  exit 1])], [$6])
 $at_traceon
 ])# _AT_CHECK