Re: CVS autoconf testsuite failure on Tru64 unix

[Noah Misch]

On Wed, Dec 15, 2004 at 02:19:48AM -0800, Noah Misch wrote:
> Now wait!  Only two Autoconf tests use constructs that look like `${...}', and
> only one of those actually expands the parameter; the other is single-quoted 
> and
> yields the syntax error that began this discussion.  It would not be harmful 
> to
> disable shell tracing for all such tests.  Yes, that will be _so_ much 
> cleaner.
> I'll post a patch separately.  Thanks for not applying this one :-)

The natural fix involved moving much of the command analysis from runtime to
autom4te-time, so I did that.  I am not certain that this patch is _simpler_
than the last patch, but I think each bit of complexity actually helps.  The old
patch was complex in unhelpful ways.

Moving analysis to autom4te-time also shrunk `testsuite' from 2748 KiB to 2022
KiB.  To be sure, I created that growth by adding all that shell code in the
first place.  For full disclosure, these are the old and new `autom4te' run
times generating `testsuite':

With this patch: 12.70user 0.21system 0:13.50elapsed 95%CPU
Before:          12.17user 0.28system 0:13.34elapsed 93%CPU

The error from run to run is about +-.05user.

My post to address@hidden about expanding text for string functions concerned
this patch.  An XFAIL test case demonstrates the lossage.  I wanted to document
it also, but no good documentation words came to mind.  If I think of some, I
will post them separately.

Thanks, Paul, for your help on this issue.

2004-12-18  Noah Misch  <address@hidden>

        * lib/autotest/general.m4 (_AT_DECIDE_TRACEABLE): New macro.
        (_AT_CHECK): Use it.
        * lib/m4sugar/m4sh.m4 (AS_ESCAPE_FOR_EXPAND): Remove.
        (AS_ESCAPE): Fix comment.
        * tests/autotest.at: Adjust section banner comments.
        (AT_CHECK_AT): Accept STATUS and STDERR.
        (AT_CHECK_AT_TEST): Likewise.
        (Invalid brace-enclosed parameter expansion)
        (Multiline command from M4 expansion)
        (Double-M4-quoted command): New tests.

diff --exclude Makefile.am -urp -X dontdiff ac-clean/lib/autotest/general.m4 
ac-simple_pexp/lib/autotest/general.m4
--- ac-clean/lib/autotest/general.m4    2004-12-14 22:30:02.000000000 -0500
+++ ac-simple_pexp/lib/autotest/general.m4      2004-12-18 02:16:29.284314403 
-0500
@@ -1187,6 +1187,110 @@ m4_define([AT_CHECK_NOESCAPE],
 [_AT_CHECK([$1],[$2],[$3],[$4],[$5],[$6])])
 
 
+# _AT_DECIDE_TRACEABLE(COMMANDS)
+# ------------------------------
+# Worker for for _AT_CHECK that expands to shell code.  If COMMANDS are safe to
+# trace with `set -x', the shell code will set `at_trace_this=yes'.  Otherwise,
+# the shell code will print a message stating an aspect of COMMANDS that makes
+# tracing them unsafe.
+# 
+# Tracing COMMANDS is not safe if they contain a command that spans multiple
+# lines.  When the test suite user passes `-x' or `--trace', the test suite
+# precedes every command with a `set -x'.  Since most tests expect a specific
+# stderr, if only to confirm that it is empty, the test suite filters ^+ from
+# the captured stderr before comparing with the expected stderr.  If a command
+# spans multiple lines, so will its trace, but a `+' only prefixes the first
+# line of that trace:
+# 
+# $ echo 'foo
+# bar'
+# => stdout
+# foo
+# bar
+# => stderr
+# + foo
+# bar
+# 
+# In a subset of cases, one could filter such extended shell traces from 
stderr.
+# Since test commands spanning several lines are rare, I chose instead to 
simply
+# not trace COMMANDS that could yield multiple trace lines.  Distinguishing 
such
+# COMMANDS became the task at hand.
+# 
+# These features may cause a shell command to span multiple lines:
+# 
+# (a) A quoted literal newline.
+# Example:
+#   echo foo'
+#   'bar
+# M4 is a hostile language for the job of parsing COMMANDS to determine whether
+# each literal newline is quoted, so we simply disable tracing for all COMMANDS
+# that bear literal newlines.
+# 
+# (b) A command substitution not subject to word splitting.
+# Example:
+#   var=$(printf 'foo\nbar')
+# Example:
+#   echo "`printf 'foo\\nbar`"
+# One cannot know in general the number of lines a command substitution will
+# yield without executing the substituted command.  As such, we disable tracing
+# for all COMMANDS containing these constructs.
+# 
+# (c) A parameter expansion not subject to word splitting.
+# Example:
+#   var=foo'
+#   'bar
+#   echo "$var"
+# Parameter expansions appear in COMMANDS with much greater frequency than do
+# newlines and command substitutions, so disabling tracing for all such 
COMMANDS
+# would much more substantially devalue `testsuite -x'.  To determine which
+# parameter expansions yield multiple lines, we escape all ``', `"', and `\' in
+# a copy of COMMANDS and expand that string within double quotes at runtime.  
If
+# the result of that expansion contains multiple lines, the test suite disables
+# tracing for the command in question.
+# 
+# This method leads the test suite to expand some parameters that the shell
+# itself will never expand due to single-quotes or backslash escapes.  This is
+# not a problem for `$foo' expansions, which will simply yield the empty string
+# or some unrelated value.  A `${...}' expansion could actually form invalid
+# shell code, however; consider `${=foo}'.  Therefore, we disable tracing for
+# all COMMANDS containing `${...}'.  This affects few COMMANDS.
+# 
+# This macro falls in a very hot path; the Autoconf test suite expands it 1640
+# times as of this writing.  To give a sense of the impact of the heuristics I
+# just described, the test suite preemptively disables tracing for 31 of those,
+# and 268 contain parameter expansions that require runtime evaluation.  The
+# balance are always safe to trace.
+# 
+# _AT_CHECK expands COMMANDS, but the Autoconf language does not provide a way
+# to safely expand arbitrary COMMANDS in an argument list, so the below tests
+# examine COMMANDS unexpanded.
+m4_define([_AT_DECIDE_TRACEABLE],
+[dnl Utility macros.
+m4_pushdef([at_reason])[]dnl
+m4_pushdef([at_lf], [
+])[]dnl
+dnl
+dnl Examine COMMANDS for a reason to never trace COMMANDS.
+m4_bmatch([$1],
+          [`.*`], [m4_pushdef([at_reason],  [a `...` command substitution])],
+          [\$(],  [m4_pushdef([at_reason], [a $(...) command substitution])],
+          [\${],  [m4_pushdef([at_reason], [a ${...} parameter expansion])],
+          at_lf,  [m4_pushdef([at_reason], [an embedded newline])])[]dnl
+dnl
+m4_ifval(m4_defn([at_reason]),
+[echo 'Not enabling shell tracing (command contains ]m4_defn([at_reason])[)'],
+[m4_bmatch([$1], [\$],
+dnl COMMANDS may contain parameter expansions; expand them an runtime.
+[case "AS_ESCAPE([$1], [`"\])" in
+        *'
+'*) echo 'Not enabling shell tracing (command contains an embedded newline)' ;;
+ *) at_trace_this=yes ;;
+    esac],
+dnl We know at build time that tracing COMMANDS is always safe.
+[at_trace_this=yes])])[]dnl
+m4_popdef([at_lf])[]dnl
+m4_popdef([at_reason])])
+
 
 # _AT_CHECK(COMMANDS, [STATUS = 0], STDOUT, STDERR,
 #           [RUN-IF-FAIL], [RUN-IF-PASS], SHELL_ESCAPE_IO)
@@ -1228,20 +1332,7 @@ echo AT_LINE >"$at_check_line_file"
 
 at_trace_this=
 if test -n "$at_traceon"; then
-    at_lf='
-'
-    at_cmd_expanded="AS_ESCAPE_FOR_EXPAND([$1])"
-    case "$at_cmd_expanded" in
-        *\$\(*\)*)         at_reason='a $(...) command substitution' ;;
-        *\`*\`*)           at_reason='a `...` command substitution' ;;
-        *"$at_lf"*)        at_reason='an embedded newline' ;;
-        *)                 at_reason= ;;
-    esac
-    if test -n "$at_reason"; then
-        echo "Not enabling shell tracing (command contains $at_reason)"
-    else
-        at_trace_this=yes
-    fi
+    _AT_DECIDE_TRACEABLE([$1])
 fi
 
 if test -n "$at_trace_this"; then
diff --exclude Makefile.am -urp -X dontdiff ac-clean/lib/m4sugar/m4sh.m4 
ac-simple_pexp/lib/m4sugar/m4sh.m4
--- ac-clean/lib/m4sugar/m4sh.m4        2004-12-13 14:46:03.000000000 -0500
+++ ac-simple_pexp/lib/m4sugar/m4sh.m4  2004-12-15 06:51:44.000000000 -0500
@@ -475,7 +475,7 @@ $as_unset $1 || test "${$1+set}" != set 
 # too.
 
 
-# AS_ESCAPE(STRING, [CHARS = $"'\])
+# AS_ESCAPE(STRING, [CHARS = $"`\])
 # ---------------------------------
 # Escape the CHARS in STRING.
 m4_define([AS_ESCAPE],
@@ -483,15 +483,6 @@ m4_define([AS_ESCAPE],
             m4_ifval([$2], [[\([$2]\)]], [[\([\"$`]\)]]),
             [\\\1])])
 
-# AS_ESCAPE_FOR_EXPAND(STRING)
-# ----------------------------
-# Escape characters in STRING that have special meaning to the shell
-# within double quotes, but leave parameter expansions active.
-# These substitutions come from sed_double_backslash in GNU Libtool.
-m4_define([AS_ESCAPE_FOR_EXPAND],
-[m4_bpatsubsts([AS_ESCAPE([$1], [`"\])],
-               [^\(\(\\\\\\\\\)*\\\\\)\$], [\1\\$],
-           [\([^\\]\(\\\\\\\\\)*\\\\\)\$], [\1\\$])])
 
 # _AS_QUOTE_IFELSE(STRING, IF-MODERN-QUOTATION, IF-OLD-QUOTATION)
 # ---------------------------------------------------------------
diff --exclude Makefile.am -urp -X dontdiff ac-clean/tests/autotest.at 
ac-simple_pexp/tests/autotest.at
--- ac-clean/tests/autotest.at  2004-12-14 22:30:02.000000000 -0500
+++ ac-simple_pexp/tests/autotest.at    2004-12-18 02:22:10.745290401 -0500
@@ -19,10 +19,13 @@ AT_BANNER([Autotest.])
 # Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA
 # 02111-1307, USA.
 
-# AT_CHECK_AT(TITLE, SUITE-CODE, [XFAIL-CONDITION])
-# -------------------------------------------------
+# AT_CHECK_AT(TITLE, SUITE-CODE, [XFAIL-CONDITION], [STATUS = 0],
+#             [STDOUT := ignore], STDERR)
+# ---------------------------------------------------------------
 # Create a new test named TITLE that runs a minimal Autotest test suite,
-# SUITE-CODE.  If given, XFAIL-CONDITION is passed to AT_XFAIL_IF.
+# SUITE-CODE.  Call AT_XFAIL_IF with XFAIL-CONDITION.  STATUS and STDERR pass
+# directly to the AT_CHECK that call the minimal test suite.  STDOUT is not
+# used, but it is reserved for future use.
 m4_define([AT_CHECK_AT],
 [
 AT_SETUP([$1])
@@ -43,15 +46,15 @@ AT_DATA([mysuite.at], [$2])
 # log file it generates will overwrite the log that the Autoconf test
 # suite produces for this test case.
 AT_CHECK_AUTOM4TE([--language=autotest -o micro-suite mysuite.at])
-AT_CHECK([./micro-suite], 0, [ignore], [])
-AT_CHECK([./micro-suite -v -x], 0, [ignore], [])
+AT_CHECK([./micro-suite],       m4_default([$4], 0), [ignore], [$6])
+AT_CHECK([./micro-suite -v -x], m4_default([$4], 0), [ignore], [$6])
 AT_CLEANUP
 ])
 
-# AT_CHECK_AT_TEST(TITLE, SUITE-SNIPPET, [XFAIL-CONDITION])
-# ---------------------------------------------------------
-# Like AT_CHECK_AT, but surrounds SUITE-SNIPPET with a boilerplate
-# AT_INIT, AT_SETUP, and AT_CLEANUP.
+# AT_CHECK_AT_TEST(TITLE, SUITE-SNIPPET, ...)
+# -----------------------------------------------------------------------
+# Wrapper for AT_CHECK_AT that surrounds SUITE-SNIPPET with a boilerplate
+# AT_INIT, AT_SETUP, and AT_CLEANUP and passes other arguments verbatim.
 m4_define([AT_CHECK_AT_TEST],
 [AT_CHECK_AT([$1],
 [[
@@ -59,14 +62,14 @@ AT_INIT([artificial test suite])
 AT_SETUP([my only test])
 $2
 AT_CLEANUP
-]], $3)])
+]], m4_shiftn(2, $@))])
 
 # Here documents for these tests contain forbidden macros.
 m4_pattern_allow([^AT_])
 
-## ------------------------------------------------------ ##
-## Newlines and command substitutions in test commandds.  ##
-## ------------------------------------------------------ ##
+## ----------------------------------------------------- ##
+## Newlines and command substitutions in test commands.  ##
+## ----------------------------------------------------- ##
 
 AT_CHECK_AT_TEST([Truth],
   [AT_CHECK([:], 0, [], [])])
@@ -111,6 +114,43 @@ bar
 bar
 ], [])])
 
+
+## ------------------------- ##
+## ${...} in test commands.  ##
+## ------------------------- ##
+
+# If this invalid parameter expansion capsizes the test suite, the entire
+# AT_SETUP ... AT_CLEANUP subshell will exit, and the commands it runs will
+# appear to have succeeded.  Therefore, we verify a failing test case.
+
+AT_CHECK_AT_TEST([Invalid brace-enclosed parameter expansion],
+  [AT_CHECK([echo '${=invalid}'], 0, [wrong])], [false], 1, ignore, ignore)
+
+
+## ---------------------------- ##
+## M4 macros in test commands.  ##
+## ---------------------------- ##
+
+# The last paragaph in the comment above _AT_DECIDE_TRACEABLE illustrates why
+# this test fails.
+AT_CHECK_AT_TEST([Multiline command from M4 expansion],
+  [m4_define([GNU], ['foo
+bar'])
+   AT_CHECK([echo GNU], 0, [foo
+bar
+], [])], [:])
+
+AT_CHECK_AT_TEST([Double-M4-quoted command],
+  [m4_define([GNU], ['foo
+bar'])
+   AT_CHECK([[echo GNU]], 0, [[GNU
+]], [])])
+
+
+## -------------------------------------- ##
+## Backslash-<newline> in test commands.  ##
+## -------------------------------------- ##
+
 AT_CHECK_AT_TEST([BS-newline in command],
   [AT_CHECK([echo Auto"\
 "conf], 0, [Autoconf