autoconf-patches
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: autotest speedups


From: Eric Blake
Subject: Re: autotest speedups
Date: Fri, 12 Oct 2007 15:36:45 +0000 (UTC)
User-agent: Loom/3.14 (http://gmane.org/)

Eric Blake <ebb9 <at> byu.net> writes:

> >> Easy: parsing.  Even with
> >>   cd autoconf/tests && ./testsuite 1
> > 
> > Cool, so my patches will indeed speed up run-time a little bit.  Or more
> > accurately, initialization time, because parsing is done only once.
> > 
> One of my ideas (although I haven't had time to try it out yet) would be
> to have the shell skip parsing the tests altogether.  Instead, emit the
> tests at the end of the file, with clear delimiters, and have the
> case/esac statement merely invoke a shell function whose job is to invoke
> sed on $0, extracting the well-labeled portion of the rest of the file
> belonging to that test into at-test, then sourcing that file.  That way,
> the shell doesn't have to parse the tests it doesn't execute.  Your patch
> may prove to be a starting point towards this idea.

I implemented the above idea with the patch below (thanks, Paolo, for the idea 
and initial work), and all I can say is:

*Wow!*

Here are some timings, on cygwin (and although I've suppressed the testsuite 
output, I verified that the tests still execute and have the same result).

pre-patch

$ time (./testsuite 1 >/dev/null)
real    0m16.485s
user    0m18.620s
sys     0m6.993s
$ time (./testsuite -10 >/dev/null)
real    1m9.188s
user    1m17.181s
sys     0m35.264s


post-patch

$ time (./testsuite 1 >/dev/null)
real    0m10.578s
user    0m11.957s
sys     0m4.503s
$ time (./testsuite -10 >/dev/null)
real    0m51.968s
user    0m58.310s
sys     0m24.852s


6 seconds (35% speedup) for executing just one test!  And when executing ten 
tests, 17 seconds speedup!  This was a bit counter-intuitive to me at first - I 
was expecting the wall-clock speedup for multiple tests to be the same or 
smaller than the speedup for one test (particularly since this version of the 
patch adds a sed run prior to every test).  So the explanation for the _bigger_ 
speedup with multiple tests must be related to a secondary benefit of my 
approach: the fewer functions you define, the less memory bash expends in 
remembering them all.  This may not matter much on Linux, with copy-on-write 
semantics; but on cygwin, where fork() involves copying the entire address 
space, less memory in use by bash equates to faster forking, and autotest is 
certainly fork-heavy.

I'm committing this patch as-is, but see a couple of potential, but competing, 
improvement ideas.

1) My patch invokes sed once per test group.  Perhaps we should instead 
generate a single sed script based on the final value of at_groups, and parse 
out all relevant shell functions with one pass of sed prior to starting the 
driver loop.  This has the benefit of fewer sed calls, but the expense of 
constant memory usage proportional to the size of the subset of tests being run.

2) Can we assume that if a shell supports functions, it also supports the 
POSIX 'unset -f'?  If so, then at_func_test should call 'unset -f 
at_func_test_$1' after executing the test, in order to free up that memory, and 
avoid cumulative slowdowns as later tests cope with larger memory footprints 
left by earlier tests.  If not, we could make all test chunks have the same 
function name, so that each subsequent `.' merely redefines that function name.
Either implementation, option 2 assumes that we keep one sed per test; it has 
the benefit of minimal memory usage per test, but the expense of more processes.

Any preferences on which option to experiment with first?

From: Eric Blake <address@hidden>
Date: Fri, 12 Oct 2007 09:29:10 -0600
Subject: [PATCH] Speed up execution of subset of testsuite.

* lib/autotest/general.m4 (TEST_FUNCTIONS): New diversion.
(AT_INIT) <at_func_test>: New shell function.
(AT_INIT) <at_myself>: New variable, set to absolute $as_myself.
(AT_INIT) <at_test_source> New variable, names file that holds
current test function definition.
(AT_SETUP): Start the shell function at_func_test_#, into the
TEST_FUNCTIONS diversion.
(AT_CLEANUP): End the shell function.  Simplify the TESTS
diversion to invoke the function.

Signed-off-by: Eric Blake <address@hidden>
---
 ChangeLog               |   14 +++++++++++
 lib/autotest/general.m4 |   57 ++++++++++++++++++++++++++++++++++++++++------
 2 files changed, 63 insertions(+), 8 deletions(-)

diff --git a/ChangeLog b/ChangeLog
index 1ccbd06..2fc0a0b 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,17 @@
+2007-10-12  Eric Blake  <address@hidden>
+       and Paolo Bonzini  <address@hidden>
+
+       Speed up execution of subset of testsuite.
+       * lib/autotest/general.m4 (TEST_FUNCTIONS): New diversion.
+       (AT_INIT) <at_func_test>: New shell function.
+       (AT_INIT) <at_myself>: New variable, set to absolute $as_myself.
+       (AT_INIT) <at_test_source> New variable, names file that holds
+       current test function definition.
+       (AT_SETUP): Start the shell function at_func_test_#, into the
+       TEST_FUNCTIONS diversion.
+       (AT_CLEANUP): End the shell function.  Simplify the TESTS
+       diversion to invoke the function.
+
 2007-10-11  Ralf Wildenhues  <address@hidden>
 
        * .gitignore: Ignore tags and TAGS files.
diff --git a/lib/autotest/general.m4 b/lib/autotest/general.m4
index cc50b5f..e34055e 100644
--- a/lib/autotest/general.m4
+++ b/lib/autotest/general.m4
@@ -105,7 +105,13 @@
 #    tail of the core for;case, overall wrap up, generation of debugging
 #    scripts and statistics.
 #  - TEST_SCRIPT
-#    The code for each test, the ``normal'' diversion
+#    The collector for code for each test, the ``normal'' diversion, but
+#    undiverted into other locations before final output.
+#
+#  - TEST_FUNCTIONS
+#    Series of functions for each test group.  The functions deliberately
+#    occur after the end of the shell script, so that the shell need not
+#    spend time parsing functions it will not execute.
 
 m4_define([_m4_divert(DEFAULTS)],           100)
 m4_define([_m4_divert(PARSE_ARGS_BEGIN)],   200)
@@ -123,6 +129,7 @@ m4_define([_m4_divert(PREPARE_TESTS)],      400)
 m4_define([_m4_divert(TESTS)],              401)
 m4_define([_m4_divert(TESTS_END)],          402)
 m4_define([_m4_divert(TEST_SCRIPT)],        403)
+m4_define([_m4_divert(TEST_FUNCTIONS)],     500)
 
 
 # AT_LINE
@@ -282,6 +289,24 @@ at_func_diff_devnull ()
   $at_diff "$at_devnull" "$[1]"
 }
 
+# at_func_test NUMBER
+# -------------------
+# Parse out at_func_test_NUMBER from the tail of this file, source it,
+# then invoke it.
+at_func_test ()
+{
+  if sed -n 
'/address@hidden:@AT_START_'$[1]$'/,/address@hidden:@AT_STOP_'$[1]$'/p' 
"$at_myself" \
+       > "$at_test_source" && . "$at_test_source" ; then
+    at_func_test_$[1] || {
+      AS_ECHO(["$as_me: unable to execute test group: $[1]"]) >&2
+      at_failed=:
+    }
+  else
+    AS_ECHO(["$as_me: unable to parse test group: $[1]"]) >&2
+    at_failed=:
+  fi
+}
+
 # Load the config file.
 for at_file in atconfig atlocal
 do
@@ -334,8 +359,14 @@ at_groups=
 
 # The directory we are in.
 at_dir=`pwd`
+# An absolute reference to this testsuite script.
+dnl m4-double quote, to preserve []
+[case $as_myself in
+  [\\/]* | ?:[\\/]* ) at_myself=$as_myself ;;
+  * ) at_myself=$at_dir/$as_myself ;;
+esac]
 # The directory the whole suite works in.
-# Should be absolutely to let the user `cd' at will.
+# Should be absolute to let the user `cd' at will.
 at_suite_dir=$at_dir/$as_me.dir
 # The file containing the suite.
 at_suite_log=$at_dir/$as_me.log
@@ -347,6 +378,8 @@ at_status_file=$at_suite_dir/at-status
 at_stdout=$at_suite_dir/at-stdout
 at_stder1=$at_suite_dir/at-stder1
 at_stderr=$at_suite_dir/at-stderr
+# The file containing the function to run a test group.
+at_test_source=$at_suite_dir/at-test-source
 # The file containing dates.
 at_times_file=$at_suite_dir/at-times
 m4_divert_pop([DEFAULTS])dnl
@@ -782,6 +815,7 @@ else
 fi
 
 
+m4_text_box([Driver loop.])
 for at_group in $at_groups
 do
   # Be sure to come back to the top test directory.
@@ -1091,6 +1125,8 @@ $at_xpass_list${at_xpass_list:+ passed unexpectedly}
 fi
 
 exit 0
+
+m4_text_box([Actual tests.])
 m4_divert_pop([TESTS_END])dnl
 dnl End of AT_INIT: divert to KILL, only test groups are to be
 dnl output, the rest is ignored.  Current diversion is BODY, inherited
@@ -1256,8 +1292,11 @@ m4_define([AT_xfail], [at_xfail=no])
 m4_define([AT_description], m4_quote($1))
 m4_define([AT_ordinal], m4_incr(AT_ordinal))
 m4_append([AT_groups_all], [ ]m4_defn([AT_ordinal]))
-m4_divert_push([TESTS])dnl
-  AT_ordinal ) @%:@ AT_ordinal. m4_defn([AT_line]): m4_defn([AT_description])
+m4_divert_push([TEST_FUNCTIONS])dnl
+[#AT_START_]AT_ordinal
address@hidden:@ AT_ordinal. m4_defn([AT_line]): m4_defn([AT_description])
+at_func_test_[]AT_ordinal ()
+{
     at_setup_line='m4_defn([AT_line])'
     at_desc="AS_ESCAPE(m4_dquote(m4_defn([AT_description])))"
     $at_quiet AS_ECHO_N([m4_format(["%3d: $at_desc%*s"], AT_ordinal,
@@ -1305,7 +1344,7 @@ m4_define([AT_CLEANUP],
 [m4_append([AT_help_all],
 m4_defn([AT_ordinal]);m4_defn([AT_line]);m4_defn([AT_description]);m4_ifdef
([AT_keywords], [m4_defn([AT_keywords])]);
 )dnl
-m4_divert_pop([TEST_SCRIPT])dnl Back to TESTS
+m4_divert_pop([TEST_SCRIPT])dnl Back to TEST_FUNCTIONS
     AT_xfail
     echo "#                             -*- compilation -*-" >> "$at_group_log"
     (
@@ -1316,9 +1355,11 @@ m4_undivert([TEST_SCRIPT])dnl Insert the code here
       $at_times_p && times >"$at_times_file"
     ) AS_MESSAGE_LOG_FD>&1 2>&1 | eval $at_tee_pipe
     at_status=`cat "$at_status_file"`
-    ;;
-
-m4_divert_pop([TESTS])dnl Back to KILL.
+}
+[#AT_STOP_]AT_ordinal
+m4_divert_pop([TEST_FUNCTIONS])dnl Back to KILL.
+m4_divert_text([TESTS],
+[  AT_ordinal ) at_func_test AT_ordinal ;;])
 ])# AT_CLEANUP
 
 
-- 
1.5.3.2







reply via email to

[Prev in Thread] Current Thread [Next in Thread]