Re: [PATCH 1/5] {test-protocols} parallel-tests: make parsing of test results safer

[Stefano Lattarini]

Hi Ralf, thanks for the tips.

On Tuesday 19 July 2011, Ralf Wildenhues wrote:
> * Stefano Lattarini wrote on Tue, Jul 19, 2011 at 11:54:09AM CEST:
> > > Please, this is really important: we need to research the other test
> > > protocols, what they do to be robust here.  Don't NIH here, because the
> > > experience we have is not enough to not mess up this.  Consider this
> > > research as part of the work needed for the SoC assignment, it is a very
> > > important part (as it is not easily corrected once released).
> > >
> > Honestly, I really don't have a clue about where to start looking for such
> > "prior art", and my rather naive Google searches haven't returned anything
> > useful.  Do you have any suggestion?  Even a loose one might be an helpful
> > starting point.
> 
> A very incomplete list:
> 
> Test protocols:
> TAP
> subunit
>
These protocols does not seems deal with the storing/registration
of testsuite outcomes (which is a good thing IMHO -- separations
of concerns and all that).

> Test suite environments:
> any that use the above
>
I see now that prove(1), the default command line interface to TAP::*
modules, has a `--state' option that allow it to store the results of
the previous testsuite run.  But the exact way this is done seems to
be an internal detail; should I look into this nonetheless?

> dejagnu
>
After having read the blog post you link below, I'd rather not look into
this at all ...

> qmtest
>
Just as an aside: the documentation of this tool seem to employ the same
terminology we are using in automake w.r.t. test results (in particular,
it uses "FAIL" and "ERROR" in the same way we do, making clear distinction
between the two); so we've probably managed to avoid NIH in this respect.

For the rest, the tool seems a little too much python-biased, but probably
we can learn something from it anyway, so its worth taking a deeper look
into.

> Test::Harness
>
I guess you mean TAP::Harness ... and yes, this would be indeed a good
place where to look too.  Silly me for not having thought about it :-(

> http://en.wikipedia.org/wiki/Portal:Software_Testing lists several
>

> Autotest
>
Hmmm... the recheck target of autotest appears to work through some
sort of "screen scraping" applied to the testsuite logs (not an example
I'm eager to follow I must say), and the format of the generated
`testsuite.log' itself seems quite ad-hoc, "grown" rather than designed.

OTOH, we should have full control on Autotest, so we might strive to make
it (mostly) compatible the Automake test harness and drivers.

> Learning from mistakes (in general):
> http://www.airs.com/blog/archives/499
> 
> Hope that helps.
> 
> Cheers,
> Ralf
> 

Thanks,
  Stefano