Re: parallel autotest [0/3]

[Ralf Wildenhues]

* Eric Blake wrote on Fri, May 30, 2008 at 12:56:14AM CEST:
> Ralf Wildenhues <Ralf.Wildenhues <at> gmx.de> writes:
> 
> > > | - (a) and (b) need GNU make for parallel execution,
> > 
> > I should clarify: (a) is likely to cope with parallel non-GNU make,
> > while (b) will ignore non-GNU make parallelism.  For example, BSD makes
> > typically allow parallel execution as well, but do not implement a job
> > server.
> 
> Yes, that helps me understand your intent - exposing the parallel tasks to 
> any 
> make's dependency engine, vs. hooking directly into the guts of GNU make's 
> job 
> server.

Exactly.

> > > Interesting trades.  I like that (c) and (d) can do parallel execution
> > > when the testsuite is run manually (without make);
> > 
> > Good point.  I would not want to have (b) without any of (c), (d);
> > rather, I thought of adding (c) and maybe also (b).
> 
> I was hoping that you could combine the good parts of (b) and (c).

Not sure what you mean by that.  My current thinking is that (b) is
kinda "neat" but strictly optional, except that it allows parallel
testsuite execution on MinGW (which does not have named pipes); without
MinGW, I would probably not propose to add it at all.

> > > The goal of still shipping a single 'testsuite' file that contains
> > > everything needed to create the multiple tests is nice.  I agree that
> > > blindly calling 'testsuite n' for n parallel tests is too much overhead.
> > > On the other hand, it would sure be nice to call testsuite once with the
> > > user's TESTSUITEFLAGS to generate the subset of individual files to run,
> > > then turn around and run those individual files in parallel.
> > 
> > I don't quite understand what the last sentence in this paragraph is
> > supposed to mean.
> 
> Our current recommended formula for including an autotest suite in Makefile 
> is 
> to call it with TESTSUITEFLAGS as the user's hook to select a subset of 
> tests.  
> So I was trying to convey my thoughts on a flow that could be something like 
> this:

Your flow amounts to something I'd list under (a), of which there are
certainly several variants resp. choices still to make.  I was kind of
hoping I didn't have to do an implementation of (a) (in order to prove
my point of it being slower), unless the current patches make us hit a
portability issue we cannot fix.

Your proposed ideas are somewhat similar to what Akim posted for
Automake's TESTS.  So there may be some insight to be gained from
looking at check.mk.

> At configure time, config.status generates testsuite and also a makefile 
> fragment that contains dependencies for each test's n.log, which is then 
> included in the user's Makefile (automake supports makefile fragment 
> inclusion 
> even for non-GNU makes, or the fragment could be a multi-line AC_SUBST rather 
> than an external file).

It doesn't have to be included in the user's Makefile.  The testsuite
could invoke make itself, too, and provide it with an explicit list of
dependencies for the set of tests chosen.  That avoids your need for a
--parallel-start.  Whether providing that explicit list amounts to
writing a Makefile at testsuite start, or just passing a make variable
to a pre-written Makefile, I don't know what's better.

But anyway, I don't see how it's better for the user to have to specify
several commands to run a testsuite.

> At make time, each n.log depends on n.test and a first stamp file.  
> Running 'testsuite --parallel-start TESTSUITEFLAGS=-10' creates the stamp 
> file, 
> creates an independent shell script n.test for tests 1-10 (each with much 
> less 
> startup overhead than testsuite; for example, it can be hardcoded to run with 
> the better shell detected by testsuite, rather than having to repeat the 
> gyrations of detecting a better shell itself), and a no-op n.test for the 
> remaining test groups (merely touching n.log).

Let's try to avoid work scaling with the number of all tests if we are
only going to run a few of them; Autotest has become pretty good at that
by now, with the remainders being:

- reading/setting the initial values of $at_groups_all, $at_help_all,
- the computation of $at_groups from the range/keyword command line
  argument,
- the sed script extracting the test group lines from the suite
  (this actually scales roughly in the size of the whole testsuite times
  the number of selected tests).

The first two are quick, and luckily, the quadratic last one has a
pretty small constant in front of it (but it's measureable overhead!).

> After the stamp file exists, 
> make can then run the various n.test files in parallel to create n.log files.
> 
> Finally, a second stamp file depends on all of the n.log files; 
> running 'testsuite --parallel-collect' gathers the results and creates the 
> second stamp.  That way, all you have to list for the dependency of 
> check-local 
> is the second stamp file.


> But that still doesn't answer the question of whether the startup costs for n 
> shell scripts is reasonable, in relation to forking off the main driver shell 
> script that has already collected a bunch of startup state.

Correct.

I'm afraid I probably won't have both time and energy to do an
implementation of (a) as you suggest, soon.  I don't mind at all if my
3/3 patch gets dropped: the timings for 2/3 and 3/3 show there's little
performance difference, and I'm not a huge proponent of adding
complexity for MinGW (and packages with multiple testsuites are
seemingly nonexistent).  If we find that patch 2/3 is fundamentally
flawed, then that would motivate me to try an implementation of (a),
otherwise I'll probably leave it up to you or somebody else to do.

> > In each of (b), (c), you can take any TESTSUITEFLAGS you would currently
> > use; with (c), add '-j3' to TESTSUITEFLAGS, with (b), add '-j3' to the 
> > gmake command line directly.
> 
> And I suppose with (b) and (c) together, you could use:
> gmake -j3 check TESTSUITEFLAGS=-j3
> 
> but then you have two job servers both competing for processor load without 
> communicating with each other on how to balance the load.

In the posted patches, the --jobserver-fds=... argument passed to the
testsuite will override the -jN argument passed to it, so there is no
ambiguity.  This priority ordering is necessary because otherwise make's
jobserver can get confused.

> > > Is there a way to write a Makefile include fragment which gets included if
> > > we detect GNU make, but is portably ignored for other makes, where we can
> > > then exploit gmake features to make parallel execution easier?
> > 
> > As I understand this question, the at_parse_makeflags snippet shown in
> > patch 3/3 does exactly that: it should be a no-op for non-GNU parallel
> > make, as none of them use '--jobserver-fds='.
> > 
> > If your question is about a general way to include something for GNU
> > make only, one possibility is to have a GNUmakefile which includes
> > Makefile plus extra, GNU make-specific code.
> 
> Whatever we decide, we should probably make it easy for package writers to 
> call 
> a single macro in configure.ac to generate the AC_SUBST that then spits out 
> all 
> the Makefile magic, rather than the current approach with several lines of 
> documentation on how to edit Makefile to add a testsuite.

Agreed.

> For that matter, it might be worth installing /bin/autotest as a thin
> shell wrapper around /bin/autom4te --language=autotest.

Agreed.

> > Fully agreed; I've seen such fishy behavior, too, but mostly ignored it
> > up to now.  I fear that the behavior depends quite a bit on the sh and
> > make implementations involved.
> 
> And I think some of it stems from the fact that we base success on the 
> contents 
> of $at_status_file, which can easily be out-of-date if a test group is killed 
> and no trap exists to update the file accordingly.

Yep.

Cheers,
Ralf