Re: [PATCH] {GSoC} parallel-tests: simplify testsuite summary

[Ralf Wildenhues]

On Thu, Jul 07, 2011 at 07:02:58PM +0200, Stefano Lattarini wrote:
> On Wednesday 06 July 2011, Ralf Wildenhues wrote:
> > But tools like autobuild have another requirement: they would like to be
> > able to detect, as reliably as possible, some statistics based on
> > whatever output is thrown at them.  IOW, they usually also would like to
> > detect the style of testsuite driver that this comes from.
> >
> But this is impossible to do reliably, as the new automake harness allows
> the use of multiple test drivers in the same Makefile.am, and even allow
> the user to define his own test drivers.  Or am I misunderstanding what
> you're saying?

No, you're not.  The art is defining something that is as reliable as
possible.

> > This is
> > easiest done if it can be done on a line by line basis.  Your proposed
> > format doesn't make that easy (but we could add a clue in one of the
> > first/early summary lines to make that easier, if we also make it easy
> > to detect the end of the summary).
> >
> Ah, now I see what you mean.  Maybe even add the "version" and "type"
> of the testsuite summary, as in:

Well, I'd hope to be able to do without a versioning of the testsuite.
That would denote a kind of failure to define a stable, easy to parse,
interface.

> > Also, if you want something extensible, then don't treat extensibility
> > like an afterthought: you could just omit any line with 0 tests, no?
> >
> Or better again add an option to do so (I personally prefer a count to be
> reported even if it is 0, but I understand that some other people might
> have a different preference).

Point is, if you add lines for new data in the future, the user has to
be resilient against them anyway.  In which case you can also just omit
the unimportant data now.  (Note some of the 0 data may be important, but
say ERROR is probably not.)

> > please show a (non-code) definition of the
> > output that would be sufficient for someone to write an (say, autobuild)
> > parser that grabs testsuite results
> >
> You mean a testsuite summary, right?  If you do, here is a rough but good
> enough definition:
> 
>   A line composed by 76 `=' characters, followed by a line beginning with
>   the string "Testsuite summary", followed again by a line composed by 76
>   `=' characters -- this marks the beginning of the testsuite results'
>   summary.  The first line composed by 76 `=' characters that comes after
>   those -- this marks the end of the testsuite results summary.  Each of
>   these results should be reported inside the testsuite summary, on a line
>   if its own, matching the following regular expression:
>    ^#\s*KIND-OF-RESULT:\s*COUNT(\s.*)?$
>   Any line in the testsuite summary that doesn't obey this format doesn't
>   denote a test result count, and should thus be ignored.

OK, let's digest this.  Assume my summary goes through email quoting, a
w32 user, gets inline-quoted in a reply.  Then /^\([|>] \)*/ should be
ignored, just as \r near EOL. '=' characters are bad because q-p will
encode them as =3D or something.  Newline mangling can easily introduce
empty lines every other line.

Then, the summary is probably not identical to another known format,
nor sufficiently easy to distinguish (which we already discussed).
Ideally, an unobtrusive way to fix that would be nice (where printing
"automake" would already count as obtrusive, as it shouldn't really
matter to the casual user who wrote this testsuite driver).

> > But clarity is only one criterion.  Let's define all criteria is has to
> > meet.
> >
> Suggestions?  I really only had "clarity" in mind :-)

Yeah, that's why I suggested stepping back and looking at it again.  ;-)

> > > Maybe this time we should clearly document the testsuite summary format?
> > 
> > Exactly.
> >
> Could you please open a bug report about that on bug-automake?  This way,
> even if we don't manage to write the documentation about this by the end
> of GSoC, we will still remember to do so for 1.12.

Will do.

> > > And now that I've been forced to spell that out, I see that the `--color'
> > > option name is poorly chosen; I'd say we change it to `--maybe-color',
> > > which is more faithful to the intended seantics.  OK?  I've done that
> > > in the attached squash-in.
> > 
> > What if the user wants color really hard?
> >
> You mean want them *even in the test-suite.log*?

Hmm.  --color is good enough, if it forces color on stdout, IMVHO.

> > Oh, is this user-influenceable at all btw?
> >
> For the testsuite summary that goes on stdout, yes, the user has complete
> control over its colorizing (as he has for the colorizing of the testsuite
> progress output); for the summary that goes in `test-suite.log', no, the
> user can't colorize it, period.  This seems to me the most sensible policy.

OK.

> > Yeah, except now the summary looks _a lot_ like output from a single
> > test.  Quite confusing for a simple parser (again, look at the autobuild
> > scripts for an example).
> >
> But IMVHO, the leading `#' characters avoids confusion for the parser, and
> the enclosing `=========...' lines avoid confusion for the humans.  Or not?

Maybe (see caveats above), but both require the "parser" to save state
between lines.  Again, this is minor, but the easier to read, the better.

> > The automake script itself uses a different method to find 'perl', not
> > 'env'.  The user can pass PERL=... to configure.  Should that be honored
> > here?
> >
> Ah, ok.  Yes, the script is always run with `$PERL -w /path/to/script' from
> within our tests.

OK then.

Thanks,
Ralf