[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: autotest speedups
From: |
Eric Blake |
Subject: |
Re: autotest speedups |
Date: |
Fri, 12 Oct 2007 15:36:45 +0000 (UTC) |
User-agent: |
Loom/3.14 (http://gmane.org/) |
Eric Blake <ebb9 <at> byu.net> writes:
> >> Easy: parsing. Even with
> >> cd autoconf/tests && ./testsuite 1
> >
> > Cool, so my patches will indeed speed up run-time a little bit. Or more
> > accurately, initialization time, because parsing is done only once.
> >
> One of my ideas (although I haven't had time to try it out yet) would be
> to have the shell skip parsing the tests altogether. Instead, emit the
> tests at the end of the file, with clear delimiters, and have the
> case/esac statement merely invoke a shell function whose job is to invoke
> sed on $0, extracting the well-labeled portion of the rest of the file
> belonging to that test into at-test, then sourcing that file. That way,
> the shell doesn't have to parse the tests it doesn't execute. Your patch
> may prove to be a starting point towards this idea.
I implemented the above idea with the patch below (thanks, Paolo, for the idea
and initial work), and all I can say is:
*Wow!*
Here are some timings, on cygwin (and although I've suppressed the testsuite
output, I verified that the tests still execute and have the same result).
pre-patch
$ time (./testsuite 1 >/dev/null)
real 0m16.485s
user 0m18.620s
sys 0m6.993s
$ time (./testsuite -10 >/dev/null)
real 1m9.188s
user 1m17.181s
sys 0m35.264s
post-patch
$ time (./testsuite 1 >/dev/null)
real 0m10.578s
user 0m11.957s
sys 0m4.503s
$ time (./testsuite -10 >/dev/null)
real 0m51.968s
user 0m58.310s
sys 0m24.852s
6 seconds (35% speedup) for executing just one test! And when executing ten
tests, 17 seconds speedup! This was a bit counter-intuitive to me at first - I
was expecting the wall-clock speedup for multiple tests to be the same or
smaller than the speedup for one test (particularly since this version of the
patch adds a sed run prior to every test). So the explanation for the _bigger_
speedup with multiple tests must be related to a secondary benefit of my
approach: the fewer functions you define, the less memory bash expends in
remembering them all. This may not matter much on Linux, with copy-on-write
semantics; but on cygwin, where fork() involves copying the entire address
space, less memory in use by bash equates to faster forking, and autotest is
certainly fork-heavy.
I'm committing this patch as-is, but see a couple of potential, but competing,
improvement ideas.
1) My patch invokes sed once per test group. Perhaps we should instead
generate a single sed script based on the final value of at_groups, and parse
out all relevant shell functions with one pass of sed prior to starting the
driver loop. This has the benefit of fewer sed calls, but the expense of
constant memory usage proportional to the size of the subset of tests being run.
2) Can we assume that if a shell supports functions, it also supports the
POSIX 'unset -f'? If so, then at_func_test should call 'unset -f
at_func_test_$1' after executing the test, in order to free up that memory, and
avoid cumulative slowdowns as later tests cope with larger memory footprints
left by earlier tests. If not, we could make all test chunks have the same
function name, so that each subsequent `.' merely redefines that function name.
Either implementation, option 2 assumes that we keep one sed per test; it has
the benefit of minimal memory usage per test, but the expense of more processes.
Any preferences on which option to experiment with first?
From: Eric Blake <address@hidden>
Date: Fri, 12 Oct 2007 09:29:10 -0600
Subject: [PATCH] Speed up execution of subset of testsuite.
* lib/autotest/general.m4 (TEST_FUNCTIONS): New diversion.
(AT_INIT) <at_func_test>: New shell function.
(AT_INIT) <at_myself>: New variable, set to absolute $as_myself.
(AT_INIT) <at_test_source> New variable, names file that holds
current test function definition.
(AT_SETUP): Start the shell function at_func_test_#, into the
TEST_FUNCTIONS diversion.
(AT_CLEANUP): End the shell function. Simplify the TESTS
diversion to invoke the function.
Signed-off-by: Eric Blake <address@hidden>
---
ChangeLog | 14 +++++++++++
lib/autotest/general.m4 | 57 ++++++++++++++++++++++++++++++++++++++++------
2 files changed, 63 insertions(+), 8 deletions(-)
diff --git a/ChangeLog b/ChangeLog
index 1ccbd06..2fc0a0b 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,17 @@
+2007-10-12 Eric Blake <address@hidden>
+ and Paolo Bonzini <address@hidden>
+
+ Speed up execution of subset of testsuite.
+ * lib/autotest/general.m4 (TEST_FUNCTIONS): New diversion.
+ (AT_INIT) <at_func_test>: New shell function.
+ (AT_INIT) <at_myself>: New variable, set to absolute $as_myself.
+ (AT_INIT) <at_test_source> New variable, names file that holds
+ current test function definition.
+ (AT_SETUP): Start the shell function at_func_test_#, into the
+ TEST_FUNCTIONS diversion.
+ (AT_CLEANUP): End the shell function. Simplify the TESTS
+ diversion to invoke the function.
+
2007-10-11 Ralf Wildenhues <address@hidden>
* .gitignore: Ignore tags and TAGS files.
diff --git a/lib/autotest/general.m4 b/lib/autotest/general.m4
index cc50b5f..e34055e 100644
--- a/lib/autotest/general.m4
+++ b/lib/autotest/general.m4
@@ -105,7 +105,13 @@
# tail of the core for;case, overall wrap up, generation of debugging
# scripts and statistics.
# - TEST_SCRIPT
-# The code for each test, the ``normal'' diversion
+# The collector for code for each test, the ``normal'' diversion, but
+# undiverted into other locations before final output.
+#
+# - TEST_FUNCTIONS
+# Series of functions for each test group. The functions deliberately
+# occur after the end of the shell script, so that the shell need not
+# spend time parsing functions it will not execute.
m4_define([_m4_divert(DEFAULTS)], 100)
m4_define([_m4_divert(PARSE_ARGS_BEGIN)], 200)
@@ -123,6 +129,7 @@ m4_define([_m4_divert(PREPARE_TESTS)], 400)
m4_define([_m4_divert(TESTS)], 401)
m4_define([_m4_divert(TESTS_END)], 402)
m4_define([_m4_divert(TEST_SCRIPT)], 403)
+m4_define([_m4_divert(TEST_FUNCTIONS)], 500)
# AT_LINE
@@ -282,6 +289,24 @@ at_func_diff_devnull ()
$at_diff "$at_devnull" "$[1]"
}
+# at_func_test NUMBER
+# -------------------
+# Parse out at_func_test_NUMBER from the tail of this file, source it,
+# then invoke it.
+at_func_test ()
+{
+ if sed -n
'/address@hidden:@AT_START_'$[1]$'/,/address@hidden:@AT_STOP_'$[1]$'/p'
"$at_myself" \
+ > "$at_test_source" && . "$at_test_source" ; then
+ at_func_test_$[1] || {
+ AS_ECHO(["$as_me: unable to execute test group: $[1]"]) >&2
+ at_failed=:
+ }
+ else
+ AS_ECHO(["$as_me: unable to parse test group: $[1]"]) >&2
+ at_failed=:
+ fi
+}
+
# Load the config file.
for at_file in atconfig atlocal
do
@@ -334,8 +359,14 @@ at_groups=
# The directory we are in.
at_dir=`pwd`
+# An absolute reference to this testsuite script.
+dnl m4-double quote, to preserve []
+[case $as_myself in
+ [\\/]* | ?:[\\/]* ) at_myself=$as_myself ;;
+ * ) at_myself=$at_dir/$as_myself ;;
+esac]
# The directory the whole suite works in.
-# Should be absolutely to let the user `cd' at will.
+# Should be absolute to let the user `cd' at will.
at_suite_dir=$at_dir/$as_me.dir
# The file containing the suite.
at_suite_log=$at_dir/$as_me.log
@@ -347,6 +378,8 @@ at_status_file=$at_suite_dir/at-status
at_stdout=$at_suite_dir/at-stdout
at_stder1=$at_suite_dir/at-stder1
at_stderr=$at_suite_dir/at-stderr
+# The file containing the function to run a test group.
+at_test_source=$at_suite_dir/at-test-source
# The file containing dates.
at_times_file=$at_suite_dir/at-times
m4_divert_pop([DEFAULTS])dnl
@@ -782,6 +815,7 @@ else
fi
+m4_text_box([Driver loop.])
for at_group in $at_groups
do
# Be sure to come back to the top test directory.
@@ -1091,6 +1125,8 @@ $at_xpass_list${at_xpass_list:+ passed unexpectedly}
fi
exit 0
+
+m4_text_box([Actual tests.])
m4_divert_pop([TESTS_END])dnl
dnl End of AT_INIT: divert to KILL, only test groups are to be
dnl output, the rest is ignored. Current diversion is BODY, inherited
@@ -1256,8 +1292,11 @@ m4_define([AT_xfail], [at_xfail=no])
m4_define([AT_description], m4_quote($1))
m4_define([AT_ordinal], m4_incr(AT_ordinal))
m4_append([AT_groups_all], [ ]m4_defn([AT_ordinal]))
-m4_divert_push([TESTS])dnl
- AT_ordinal ) @%:@ AT_ordinal. m4_defn([AT_line]): m4_defn([AT_description])
+m4_divert_push([TEST_FUNCTIONS])dnl
+[#AT_START_]AT_ordinal
address@hidden:@ AT_ordinal. m4_defn([AT_line]): m4_defn([AT_description])
+at_func_test_[]AT_ordinal ()
+{
at_setup_line='m4_defn([AT_line])'
at_desc="AS_ESCAPE(m4_dquote(m4_defn([AT_description])))"
$at_quiet AS_ECHO_N([m4_format(["%3d: $at_desc%*s"], AT_ordinal,
@@ -1305,7 +1344,7 @@ m4_define([AT_CLEANUP],
[m4_append([AT_help_all],
m4_defn([AT_ordinal]);m4_defn([AT_line]);m4_defn([AT_description]);m4_ifdef
([AT_keywords], [m4_defn([AT_keywords])]);
)dnl
-m4_divert_pop([TEST_SCRIPT])dnl Back to TESTS
+m4_divert_pop([TEST_SCRIPT])dnl Back to TEST_FUNCTIONS
AT_xfail
echo "# -*- compilation -*-" >> "$at_group_log"
(
@@ -1316,9 +1355,11 @@ m4_undivert([TEST_SCRIPT])dnl Insert the code here
$at_times_p && times >"$at_times_file"
) AS_MESSAGE_LOG_FD>&1 2>&1 | eval $at_tee_pipe
at_status=`cat "$at_status_file"`
- ;;
-
-m4_divert_pop([TESTS])dnl Back to KILL.
+}
+[#AT_STOP_]AT_ordinal
+m4_divert_pop([TEST_FUNCTIONS])dnl Back to KILL.
+m4_divert_text([TESTS],
+[ AT_ordinal ) at_func_test AT_ordinal ;;])
])# AT_CLEANUP
--
1.5.3.2
- autotest speedups, Eric Blake, 2007/10/04
- Re: autotest speedups, Paolo Bonzini, 2007/10/04
- Re: autotest speedups, Ralf Wildenhues, 2007/10/08
- Re: autotest speedups, Paolo Bonzini, 2007/10/09
- Re: autotest speedups, Eric Blake, 2007/10/11
- Re: autotest speedups,
Eric Blake <=
- Re: autotest speedups, Paolo Bonzini, 2007/10/12
- Re: autotest speedups, Ralf Wildenhues, 2007/10/12
- Re: autotest speedups, Eric Blake, 2007/10/12
Re: autotest speedups, Paolo Bonzini, 2007/10/04