Re: Autotest function usage

[Ralf Wildenhues]

* Eric Blake wrote on Thu, Nov 15, 2007 at 04:25:00AM CET:
> Timings on Cygwin; best of three (except for the time-consuming -100 test).
> 
> Pre patch:
> $ \time ./testsuite 1
> 14.26user 6.09system 0:14.08elapsed 144%CPU (0avgtext+0avgdata

> $ \time ./testsuite -10
> 69.20user 27.79system 1:06.96elapsed 144%CPU (0avgtext+0avgdata

> $ \time ./testsuite -100
> 1223.11user 495.20system 18:54.70elapsed 151%CPU (0avgtext+0avgdata

> Post patch:
> $ \time ./testsuite 1
> 13.66user 5.35system 0:13.42elapsed 141%CPU (0avgtext+0avgdata

> $ \time ./testsuite -10
> 66.45user 28.01system 1:10.71elapsed 133%CPU (0avgtext+0avgdata

> $ \time ./testsuite -100
> 1162.81user 488.71system 18:14.99elapsed 150%CPU (0avgtext+0avgdata

> Cygwin gained 4% on both a single test and 10 tests, and gained 5.9% on
> the 100 tests (although that number is a bit skewed, since many of the
> first 100 tests include a microsuite that runs a single test, so you get
> some of the effects of the 1-test speedup or penalty for each of tests
> 53-100).

Looks to me like the patch regresses on 10 tests and improves on 1 and
100, rather than an improvement everywhere.  Since the improvement on
100 tests is quite noticable, I guess that's ok.

> > Do you think this is worthwhile to apply?
> 
> Yes, after the nits in my previous mail are addressed.  (A shell script
> that emits an awk script that emits a shell script that creates files for
> sourcing, all because it is faster than making the original shell do all
> the parsing.  What an interesting concept.)

Yep.

> > +# Create a test file that has more than 99 words in a line, for Solaris 
> > awk.
> > +# While at that, try out the limit of 2000 bytes in a text file line.
> > +
> > +AT_CHECK_AT_TEST([Long test source lines],
> > +[m4_for([nnn], [1], [999], [], [: ])
> > +AT_CHECK([:])
> > +])
> 
> Cute.

Well, I checked again, this test isn't quite yet what I wanted.  I would
like to prevent us from accidentally inducing awk to split the input.
Right now, there would be no need for a test to this end: Autoconf's
testsuite itself will fail to be parsed correctly by awk due to the
| at_groups_all=' 1 2 3 ... '

line which has many more than 99 words.  However, this part of the code
is due for a rewrite soon anyway, as it's bound to exceed portable text
file line length eventually.

The `Long test source lines' will keep us from regressing even then.
But for that, let's also get the awk script to run over the test in the
non-emit state: run it with the test disabled (-k skipalltests).

So I applied like this.

Cheers,
Ralf

2007-11-15  Paolo Bonzini  <address@hidden>
        and Ralf Wildenhues  <address@hidden>

        * lib/autotest/general.m4 (at_func_test): Remove.
        (AT_INIT): Pre-extract test groups into separate files.
        (AT_CLEANUP): Source pre-extracted file instead of calling at_func_test.
        Remove at-test-source files together with the $at_group_dir.
        * tests/autotest.at (Long test source lines): New test.

diff --git a/lib/autotest/general.m4 b/lib/autotest/general.m4
index 4d92e6d..fd39ae9 100644
--- a/lib/autotest/general.m4
+++ b/lib/autotest/general.m4
@@ -185,6 +185,7 @@ m4_define([_AT_NORMALIZE_TEST_GROUP_NUMBER],
 # Begin test suite.
 m4_define([AT_INIT],
 [m4_pattern_forbid([^_?AT_])
+m4_pattern_allow([^_AT_T_EOF$])
 m4_define([AT_TESTSUITE_NAME],
          m4_defn([AT_PACKAGE_STRING])[ test suite]m4_ifval([$1], [: $1]))
 m4_define([AT_ordinal], 0)
@@ -310,15 +311,6 @@ at_func_diff_devnull ()
   $at_diff "$at_devnull" "$[1]"
 }
 
-# at_func_test NUMBER
-# -------------------
-# Parse out test NUMBER from the tail of this file.
-at_func_test ()
-{
-  sed -n 
'/address@hidden:@AT_START_'$[1]'$/,/address@hidden:@AT_STOP_'$[1]'$/p' 
"$at_myself" \
-       > "$at_test_source"
-}
-
 # at_func_create_debugging_script
 # -------------------------------
 # Create the debugging script $at_group_dir/run which will reproduce the
@@ -431,7 +423,7 @@ at_status_file=$at_suite_dir/at-status
 at_stdout=$at_suite_dir/at-stdout
 at_stder1=$at_suite_dir/at-stder1
 at_stderr=$at_suite_dir/at-stderr
-# The file containing the function to run a test group.
+# The stem for files containing a test group.
 at_test_source=$at_suite_dir/at-test-source
 # The file containing dates.
 at_times_file=$at_suite_dir/at-times
@@ -871,6 +863,36 @@ else
   at_diff=diff
 fi
 
+{
+  echo 'BEGIN {'
+  for at_group in $at_groups; do
+    at_group_normalized=$at_group
+    _AT_NORMALIZE_TEST_GROUP_NUMBER(at_group_normalized)
+    echo "  outfile[[\"$at_group\"]] = 
\"$at_test_source-$at_group_normalized\""
+  done
+  AS_ECHO(['}
+emit == 0 && /address@hidden:@AT_START_/ {
+  test = substr($ 0, 11);
+  if (outfile[[test]]) {
+    emit = 1
+    print "cat >\"" outfile[[test]] "\" <<'\''_AT_T_EOF'\''"
+  }
+}
+emit != 0 && /address@hidden:@AT_STOP_/ {
+  print "_AT_T_EOF"
+  emit = 0
+}
+emit != 0 { print }
+'])
+} > "$at_test_source.awk"
+
+# Extract test group that will be run from the tail of this file
+if awk -f "$at_test_source.awk" "$at_myself" >"$at_test_source.sh" \
+   && . "$at_test_source.sh" \
+   && rm -f "$at_test_source.awk" "$at_test_source.sh"; then :; else
+  AS_ECHO(["$as_me: unable to parse test groups"]) >&2
+  exit 1
+fi
 
 m4_text_box([Driver loop.])
 for at_group in $at_groups
@@ -906,10 +928,7 @@ do
     at_tee_pipe='cat >> "$at_group_log"'
   fi
 
-  if at_func_test $at_group && . "$at_test_source"; then :; else
-    AS_ECHO(["$as_me: unable to parse test group: $at_group"]) >&2
-    at_failed=:
-  fi
+  . "$at_test_source-$at_group_normalized"
 
   # Be sure to come back to the suite directory, in particular
   # since below we might `rm' the group directory we are in currently.
@@ -973,9 +992,12 @@ _ATEOF
       # Cleanup the group directory, unless the user wants the files.
       if $at_debug_p ; then
        at_func_create_debugging_script
-      elif test -d "$at_group_dir"; then
-       find "$at_group_dir" -type d ! -perm -700 -exec chmod u+rwx \{\} \;
-       rm -fr "$at_group_dir"
+      else
+       if test -d "$at_group_dir"; then
+         find "$at_group_dir" -type d ! -perm -700 -exec chmod u+rwx \{\} \;
+         rm -fr "$at_group_dir"
+        fi
+       rm -f "$at_test_source-$at_group_normalized"
       fi
       ;;
     *)
diff --git a/tests/autotest.at b/tests/autotest.at
index 54e7464..fa7b7ea 100644
--- a/tests/autotest.at
+++ b/tests/autotest.at
@@ -335,6 +335,21 @@ AT_CHECK_AT_TITLE_CHAR([Longer test title],
                       [01234567890123456789012345678901234], [], [], [54])
 
 
+## ----------------------- ##
+## Long test source lines. ##
+## ----------------------- ##
+
+# Create a test file that has more than 99 words in a line, for Solaris awk.
+# While at that, try out the limit of 2000 bytes in a text file line.
+
+AT_CHECK_AT_TEST([Long test source lines],
+[m4_for([nnn], [1], [999], [], [: ])
+AT_CHECK([:])
+], [], [], [], [ignore],
+[AT_CHECK([$CONFIG_SHELL ./micro-suite -k skipalltests], [], [ignore], 
[ignore])
+])
+
+
 ## ----------------- ##
 ## Debugging a test. ##
 ## ----------------- ##