[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: parallel autotest [0/3]
Re: parallel autotest [0/3]
Thu, 29 May 2008 22:56:14 +0000 (UTC)
Ralf Wildenhues <Ralf.Wildenhues <at> gmx.de> writes:
> > |
> > | - (a) and (b) need GNU make for parallel execution,
> I should clarify: (a) is likely to cope with parallel non-GNU make,
> while (b) will ignore non-GNU make parallelism. For example, BSD makes
> typically allow parallel execution as well, but do not implement a job
Yes, that helps me understand your intent - exposing the parallel tasks to any
make's dependency engine, vs. hooking directly into the guts of GNU make's job
> > | - AFAICS (b) currently needs a shell that understands 'read -n1',
> > | - (d) differs from (c) in that one subprocess executes more than one
> > | test group, thus is potentially faster because it forks less,
> > | - (b) has the nice feature that it allows to parallelize across multiple
> > | test suites, and across testing and other, independent build activity.
> > | That means, while (c) and (d) allow
> > | make check TESTSUITEFLAGS='-j3'
> > | to speed up things, (b) allows
> > | gmake -j3 check
> > | to profit.
> > Interesting trades. I like that (c) and (d) can do parallel execution
> > when the testsuite is run manually (without make);
> Good point. I would not want to have (b) without any of (c), (d);
> rather, I thought of adding (c) and maybe also (b).
I was hoping that you could combine the good parts of (b) and (c).
> > The goal of still shipping a single 'testsuite' file that contains
> > everything needed to create the multiple tests is nice. I agree that
> > blindly calling 'testsuite n' for n parallel tests is too much overhead.
> > On the other hand, it would sure be nice to call testsuite once with the
> > user's TESTSUITEFLAGS to generate the subset of individual files to run,
> > then turn around and run those individual files in parallel.
> I don't quite understand what the last sentence in this paragraph is
> supposed to mean.
Our current recommended formula for including an autotest suite in Makefile is
to call it with TESTSUITEFLAGS as the user's hook to select a subset of tests.
So I was trying to convey my thoughts on a flow that could be something like
At configure time, config.status generates testsuite and also a makefile
fragment that contains dependencies for each test's n.log, which is then
included in the user's Makefile (automake supports makefile fragment inclusion
even for non-GNU makes, or the fragment could be a multi-line AC_SUBST rather
than an external file).
At make time, each n.log depends on n.test and a first stamp file.
Running 'testsuite --parallel-start TESTSUITEFLAGS=-10' creates the stamp file,
creates an independent shell script n.test for tests 1-10 (each with much less
startup overhead than testsuite; for example, it can be hardcoded to run with
the better shell detected by testsuite, rather than having to repeat the
gyrations of detecting a better shell itself), and a no-op n.test for the
remaining test groups (merely touching n.log). After the stamp file exists,
make can then run the various n.test files in parallel to create n.log files.
Finally, a second stamp file depends on all of the n.log files;
running 'testsuite --parallel-collect' gathers the results and creates the
second stamp. That way, all you have to list for the dependency of check-local
is the second stamp file.
But that still doesn't answer the question of whether the startup costs for n
shell scripts is reasonable, in relation to forking off the main driver shell
script that has already collected a bunch of startup state.
> In each of (b), (c), you can take any TESTSUITEFLAGS you would currently
> use; with (c), add '-j3' to TESTSUITEFLAGS, with (b), add '-j3' to the
> gmake command line directly.
And I suppose with (b) and (c) together, you could use:
gmake -j3 check TESTSUITEFLAGS=-j3
but then you have two job servers both competing for processor load without
communicating with each other on how to balance the load.
> > Is there a way to write a Makefile include fragment which gets included if
> > we detect GNU make, but is portably ignored for other makes, where we can
> > then exploit gmake features to make parallel execution easier?
> As I understand this question, the at_parse_makeflags snippet shown in
> patch 3/3 does exactly that: it should be a no-op for non-GNU parallel
> make, as none of them use '--jobserver-fds='.
> If your question is about a general way to include something for GNU
> make only, one possibility is to have a GNUmakefile which includes
> Makefile plus extra, GNU make-specific code.
Whatever we decide, we should probably make it easy for package writers to call
a single macro in configure.ac to generate the AC_SUBST that then spits out all
the Makefile magic, rather than the current approach with several lines of
documentation on how to edit Makefile to add a testsuite. For that matter, it
might be worth installing /bin/autotest as a thin shell wrapper
around /bin/autom4te --language=autotest.
> Fully agreed; I've seen such fishy behavior, too, but mostly ignored it
> up to now. I fear that the behavior depends quite a bit on the sh and
> make implementations involved.
And I think some of it stems from the fact that we base success on the contents
of $at_status_file, which can easily be out-of-date if a test group is killed
and no trap exists to update the file accordingly.
> I think 1.5.25. My usual procedure is to boot w32, fire up cygwin
> setup.exe, run its update process without changing any settings
> manually, then do testing. I assume that, for experimental 1.7.0
> I'd have to change some setting, no?
Your guess is right - since you haven't done anything special, you're running