Re: autotest: --keywords, and test titles

[Ralf Wildenhues]

Hi Paul,

Thanks for the review!

* Paul Eggert wrote on Fri, Mar 10, 2006 at 12:37:22AM CET:
> Ralf Wildenhues <address@hidden> writes:
> 
> > Would `!' be a good character for negation?
> 
> '-' might be better if you're worried about quoting issues.

Hmm, yes, but I was a bit wary of that.  It would require us to parse
  -k -keyword

right.  Now we get that right _at the moment_, but who knows whether
someone will suggest getopt-like parsing sometime, where this may be
ambiguous or at least harder to do right.

I thought about using '^' instead, but that has a different meaning both
in regex context, as well as in logic operation context, so it's not too
intuitive.  It has the advantage that it needs to be quoted in very few
shells only, though.

I'm undecided, really.

> > -  at_optarg=`expr "x$at_option" : 'x[[^=]]*=\(.*\)'`
> > +  case $at_option in
> > +  *=*) at_optarg=`expr "x$at_option" : 'x[[^=]]*=\(.*\)'` ;;
> > +  *)   at_optarg= ;;
> > +  esac
> 
> Is this change for efficiency only, or is something deeper going on?

Both.  I did this for efficiency, but I would not be surprised if QNX
expr (which I do not have access to) would output 0 if it found no `=',
given the portability warnings in the Autoconf manual.  We can omit it
if you prefer, it's not a hot code path.

### Efficiency side note ###
In CVS Libtool we both assume shell function support and we detect at
`configure' time whether the shell has some Posix support.  We then use
that to emit either fast or slow fallback functions.  I'd have to test
whether this can also be portably done in-place (i.e., to make use of it
even at configure time).  I think this is possible, though, like this:

# ac_func_skip_first_sep INPUT SEPARATOR
# remove everything up to the first SEPARATOR from INPUT.
# SEPARATOR may not be regex active nor shell globbing active,
# [ For the corresponding postfix function, disallow '%' as well. ]
if $ac_posix_shell; then
  eval '
  ac_func_skip_first_sep () {
    case $1 in
    *"$2"*) ac_result=$1                  # work around pdksh bug
            ac_result=${ac_result#*$2} ;;
    *)      ac_result= ;;
    esac
  }'
else
  ac_func_skip_first_sep () {
    case $1 in
    *"$2"*) ac_result=`expr "x$1" : "x[^$2]*$2\(.*\)"` ;;
    *)      ac_result= ;;
    esac
  }
fi

The uglyness of using $ac_result in the mainline code buys a fast path
without any forks, while keeping compatible even with ancient shells, I
believe; the eval is to keep them from barfing over bad syntax, and the
pdksh bug is that constructs like `${1#$2}' are not evaluated correctly
if both operands are positional parameters (independent of the modifier,
be that %, #, %%, or ##).  The calculation of $ac_posix_shell could rule
out further bugs.

Just thinking out loud for Autoconf 3.0, by the way.. about a dozen of
these functions should be sufficient to kill a sizeable fraction of
forks from typical configure scripts.  And it's possible to move over
to this incrementally, which is really nice.  :-)

### End of side note ###

> > +       at_keyword=`expr "X$at_keyword" : '..\(.*\)'`
> 
> It would be a bit clearer to change '..\(.*\)' to 'X!\(.*\)'.

Agreed.

Meanwhile, I found what the real bug was that prevented 2.59 autotest
to avoid running tests more than once with
  ./testsuite 1-3 2-4
namely a couple of simple typos: `at_fail_tests' instead of
`at_fail_list' etc.  That code can simply go now: I like Stepan's
involved solution much better, with its nice sorted execution and
banner insertion.  ;-)

Proposed updated patch, including documentation and tests below.

Cheers,
Ralf

        * lib/autotest/general.m4 (AT_INIT) <at_optarg>: Optimize
        `expr' away if there is nothing to do.
        < --keywords >: Simplify and robustify argument handling.
        Revert erroneous comment from 2005-08-23.  Extend to allow
        keyword negation with `!'.
        Update help message.  Remove broken code to prevent running
        tests multiple times.
        * doc/autoconf.texi (testsuite Invocation) < --keywords >:
        Update and fix the documentation accordingly.
        * tests/autotest.at (Keywords): Renamed to..
        (Keywords and ranges): .. this.  Extended to make sure negated
        keywords, keywords taken from AT_SETUP arguments, and numeric
        test ranges work, and that matching is case-insensitive.

Index: doc/autoconf.texi
===================================================================
RCS file: /cvsroot/autoconf/autoconf/doc/autoconf.texi,v
retrieving revision 1.962
diff -u -r1.962 autoconf.texi
--- doc/autoconf.texi   11 Mar 2006 13:15:55 -0000      1.962
+++ doc/autoconf.texi   11 Mar 2006 17:11:42 -0000
@@ -16725,13 +16725,29 @@
 @itemx -k @var{keywords}
 Add to the selection the test groups with title or keywords (arguments
 to @code{AT_SETUP} or @code{AT_KEYWORDS}) that match @emph{all} keywords
-of the comma separated list @var{keywords}.
+of the comma separated list @var{keywords}, case-insensitively.  Use
address@hidden immediately before the keyword to invert the selection for this
+keyword.  By default, the keywords match whole words; enclose them in
address@hidden to also match parts of words.
 
-Running @samp{./testsuite -k autoupdate,FUNC} will select all the tests
-tagged with @samp{autoupdate} @emph{and} @samp{FUNC} (as in
address@hidden, @samp{AC_FUNC_FNMATCH}, etc.)@: while
address@hidden/testsuite -k autoupdate -k FUNC} will select all tests tagged 
with
address@hidden @emph{or} @samp{FUNC}.
+For example, running
+
address@hidden
address@hidden/testsuite -k 'autoupdate,.*FUNC.*'}
address@hidden example
+
address@hidden
+will select all tests tagged @samp{autoupdate} @emph{and} with tags
+containing @samp{FUNC} (as in @samp{AC_CHECK_FUNC}, @samp{AC_FUNC_FNMATCH},
+etc.), while
+
address@hidden
address@hidden/testsuite -k '!autoupdate' -k '.*FUNC.*'}
address@hidden example
+
address@hidden
+will select all tests not tagged @samp{autoupdate} @emph{or} with tags
+containing @samp{FUNC}.
 
 @item --errexit
 @itemx -e
Index: lib/autotest/general.m4
===================================================================
RCS file: /cvsroot/autoconf/autoconf/lib/autotest/general.m4,v
retrieving revision 1.197
diff -u -r1.197 general.m4
--- lib/autotest/general.m4     5 Mar 2006 08:29:50 -0000       1.197
+++ lib/autotest/general.m4     11 Mar 2006 17:11:43 -0000
@@ -83,7 +83,7 @@
 #    Modes help text.  Additional modes can be appended as self-contained
 #    cat'd here-docs as generated by AS_HELP_STRING.
 #  - HELP_TUNING
-#    TUning help text.  Additional tuning options can be appended as
+#    Tuning help text.  Additional tuning options can be appended as
 #    self-contained cat'd here-docs as generated by AS_HELP_STRING.
 #  - HELP_OTHER
 #    User help can be appended to this as self-contained cat'd here-docs.
@@ -273,7 +273,10 @@
     at_prev=
   fi
 
-  at_optarg=`expr "x$at_option" : 'x[[^=]]*=\(.*\)'`
+  case $at_option in
+  *=*) at_optarg=`expr "x$at_option" : 'x[[^=]]*=\(.*\)'` ;;
+  *)   at_optarg= ;;
+  esac
 
   # Accept the important Cygnus configure options, so we can diagnose typos.
 
@@ -358,11 +361,23 @@
        ;;
     --keywords=* )
        at_groups_selected=$at_help_all
-       for at_keyword in `IFS=,; set X $at_optarg; shift; echo address@hidden
+       at_save_IFS=$IFS
+       IFS=,
+       set X $at_optarg
+       shift
+       IFS=$at_save_IFS
+       for at_keyword
        do
-               # Do not match the test group titles.
-               at_groups_selected=`echo "$at_groups_selected" |
-                       grep -i ["^[1-9][^;]*;.*[; ]$at_keyword[ ;]"]`
+         at_invert=
+         case $at_keyword in
+         '!'*)
+           at_invert="-v"
+           at_keyword=`expr "X$at_keyword" : 'X!\(.*\)'`
+           ;;
+         esac
+         # It is on purpose that we match the test group titles too.
+         at_groups_selected=`echo "$at_groups_selected" |
+             grep -i $at_invert ["^[1-9][^;]*;.*[; ]$at_keyword[ ;]"]`
        done
        at_groups_selected=`echo "$at_groups_selected" | sed 's/;.*//'`
        # Smash the newlines.
@@ -429,8 +444,8 @@
   cat <<_ATEOF
 Usage: $[0] [[OPTION]... [VARIABLE=VALUE]... [TESTS]]
 
-Run all the tests, or the selected TESTS, and save a detailed log file.
-Upon failure, create debugging scripts.
+Run all the tests, or the selected TESTS, given by numeric ranges, and
+save a detailed log file.  Upon failure, create debugging scripts.
 
 You should not change environment variables unless explicitly passed
 as command line arguments.  Set \`AUTOTEST_PATH' to select the executables
@@ -459,8 +474,8 @@
 
 Execution tuning:
   -k, --keywords=KEYWORDS
-                select the tests matching all the comma separated KEYWORDS
-                accumulates
+                select the tests matching all the comma-separated KEYWORDS
+                multiple \`-k' accumulate; prefixed \`!' negates a KEYWORD
   -e, --errexit  abort as soon as a test fails; implies --debug
   -v, --verbose  force more detailed output
                 default for debugging scripts
@@ -674,12 +689,6 @@
       ;;
 
     *)
-      # Skip tests we already run (using --keywords makes it easy to get
-      # duplication).
-      case " $at_pass_test $at_skip_test $at_fail_test " in
-       *" $at_group "* ) continue;;
-      esac
-
       at_group_normalized=$at_group
       _AT_NORMALIZE_TEST_GROUP_NUMBER(at_group_normalized)
 
@@ -1159,7 +1168,7 @@
 
 
 # AT_XFAIL_IF(SHELL-EXPRESSION)
-# -----------------------------------
+# -----------------------------
 # Set up the test to be expected to fail if SHELL-EXPRESSION evaluates to
 # true (exitcode = 0).
 m4_define([AT_XFAIL_IF],
Index: tests/autotest.at
===================================================================
RCS file: /cvsroot/autoconf/autoconf/tests/autotest.at,v
retrieving revision 1.8
diff -u -r1.8 autotest.at
--- tests/autotest.at   23 Aug 2005 08:57:24 -0000      1.8
+++ tests/autotest.at   11 Mar 2006 17:11:43 -0000
@@ -2,7 +2,7 @@
 
 AT_BANNER([Autotest.])
 
-# Copyright (C) 2004, 2005 Free Software Foundation, Inc.
+# Copyright (C) 2004, 2005, 2006 Free Software Foundation, Inc.
 #
 # This program is free software; you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -246,7 +246,7 @@
 ## --------- ##
 ## Keywords. ##
 ## --------- ##
-AT_SETUP([Keywords])
+AT_SETUP([Keywords and ranges])
 AT_KEYWORDS([autotest])
 
 AT_DATA([k.at],
@@ -289,4 +289,31 @@
 AT_CHECK_KEYS([-k key2], [second|both], [2], [none|first], [0])
 AT_CHECK_KEYS([-k key1,key2], [both], [1], [none|first|second], [0])
 AT_CHECK_KEYS([-k key1 -k key2], [first|second|both], [3], [none], [0])
+AT_CHECK_KEYS([-k '!key1'], [none|second], [2], [first|both], [0])
+AT_CHECK_KEYS([-k '!key2'], [none|first], [2], [second|both], [0])
+AT_CHECK_KEYS([-k '!key1,key2'], [second], [1], [none|first|both], [0])
+AT_CHECK_KEYS([-k 'key1,!key2'], [first], [1], [none|second|both], [0])
+AT_CHECK_KEYS([-k '!key1,!key2'], [none], [1], [first|second|both], [0])
+AT_CHECK_KEYS([-k '!key1' -k KEY2], [none|second|both], [3], [first], [0])
+AT_CHECK_KEYS([-k key1 -k '!key2'], [none|first|both], [3], [second], [0])
+AT_CHECK_KEYS([-k '!KEY1' -k '!key2'], [none|first|second], [3], [both], [0])
+
+AT_CHECK_KEYS([-k none], [none], [1], [first|second|both], [0])
+AT_CHECK_KEYS([-k key1,both], [both], [1], [none|first|second], [0])
+AT_CHECK_KEYS([-k key1 -k both], [first|both], [2], [none|second], [0])
+AT_CHECK_KEYS([-k none,first], [successful], [1], [none|first|second|both], 
[0])
+AT_CHECK_KEYS([-k none,first,second,both], [successful], [1], 
[none|first|second|both], [0])
+AT_CHECK_KEYS([-k !none,first], [first], [1], [none|second|both], [0])
+
+AT_CHECK_KEYS([-k '.*eco.*'], [second], [1], [none|first|both], [0])
+AT_CHECK_KEYS([-k 'ECO'], [successful], [1], [none|first|second|both], [0])
+AT_CHECK_KEYS([-k '.*eco'], [successful], [1], [none|first|second|both], [0])
+AT_CHECK_KEYS([-k 'eco.*'], [successful], [1], [none|first|second|both], [0])
+AT_CHECK_KEYS([-k 'fir.*'], [first], [1], [none|second|both], [0])
+
+AT_CHECK_KEYS([1-2], [none|first], [2], [second|both], [0])
+AT_CHECK_KEYS([1-3 2-1], [none|first|second], [3], [both], [0])
+AT_CHECK_KEYS([-3], [none|first|second], [3], [both], [0])
+AT_CHECK_KEYS([4-], [both], [1], [none|first|second], [0])
+AT_CHECK_KEYS([-k second 4-], [second|both], [2], [none|first], [0])
 AT_CLEANUP