Re: [Tsp-devel] A test campaign manager for dtest

[Frederik Deweerdt]

On Fri, Apr 11, 2008 at 11:01:18PM +0200, Eric Noulard wrote:
> 2008/4/11, Frederik Deweerdt <address@hidden>:
> >  > The attached file details the main goals of the project. Please let us 
> > know
> >  > if you are interested in such an evolution inside dtest. Your potential
> >  > remarks and/or propositions would of course be greatly appreciated.
> >
> > Nice proposal, thanks!
> 
> +1
> 
> >
> >  I've got a few questions/remarks:
> >
> >  - IMHO, this should be part of the dtest distribution
> 
> I agree to but:
> 
> I want to keep DTest as simple as possible.
> 
> For example using DTest does not imply using a Test Campagn Manager
> DTest should be nicely layered such that lower blocks
> may be re-used.
> 
> >  - Would'nt YAML[1] be more flexible than XML? e.g. :
> >
> [...]
> 
> >  this would lessen the tests's burden of parameter parsing. We could in
> >  particular avoid:
> >  <parameter name="client_ip1" value="2.2.2.2"/>
> >  <parameter name="client_ip2" value="2.2.2.2"/>
> 
> I've no precise idea about that [yet]
> 
> >  - How does the campaign know about a particular test failing? Do we rely
> >   on forwarding the test's TAP output?
> 
> We discussed the fact that "DTest report" may be of different kind
> TAP was the first, Lionel already added MSC-like output.
I understand that most of us have no use of TAP output, but why not use
TAP-to-AnythingElse translators, that parse the DTestMaster's output,
so that inside dtest, we always use TAP?

> Independently of the "output format" each DTester should (generically)
> elaborate a test result. If an ok step has failed then DTester has failed
> then DTestmaster know it then Campaign Manager too.
> 
> > In that case, the output needs to be
> >   serialized it if several clients are running at the same time.
> 
> Unless I miss your point I think DTestmaster already serialize
> the "distributed test" output?
> 
> In fact there is no "real" serialize need since DTestmaster and
> all DTester runs on the same machine and are LOCALLY synchronized.
> However DTester have asynchronous "output" from the SSH Session Handler.
The case I was thinking of is as follows:

D Campaign Manager -> DTestMaster -> DTest sibling
   Understands TAP <- TAP         <- TAP
So it would make sense that the Campaign Manager sees:
TAP from client 1
TAP from client 1
TAP from client 1
TAP from client 2
TAP from client 2
TAP from client 2
Eventhough the lines where not received in that order by the
DTestMaster.
> 
> The tests sequence itself may/should contains expect/barrier steps
> which ensure synchronization when (functionnalmy) needed.
Sure, my point is that the output needs to be serialized so that it's
easy to parse the TAP at the other end.
> 
> I will be silent for a week, since I may not touch a keyboard during
> this time :=)
Have a nice time then!

Frederik