Re: [Tsp-devel] A test campaign manager for dtest

[Eric Noulard]

2008/4/14, Frederik Deweerdt <address@hidden>:
> On Fri, Apr 11, 2008 at 11:01:18PM +0200, Eric Noulard wrote:
>  > >  - How does the campaign know about a particular test failing? Do we rely
>  > >   on forwarding the test's TAP output?
>  >
>  > We discussed the fact that "DTest report" may be of different kind
>  > TAP was the first, Lionel already added MSC-like output.
>
> I understand that most of us have no use of TAP output, but why not use
>  TAP-to-AnythingElse translators, that parse the DTestMaster's output,
>  so that inside dtest, we always use TAP?

I think we cannot only use TAP inside DTest since TAP is too "poor"
for some purpose.
TAP is [very]-good for simple go-nogo test, but I want to
be able to implement more elaborate "passive testing" mechanism.
see eg: http://www-lor.int-evry.fr/~anna/passive_testing_revue.pdf

Practically TAP only gives us vision of the "ok" steps,
I want to be able to trace ALL steps: ok, barrier, expect, send, ....
As an example I may want to define something like an "observer step"
which may gives me [non-intrusively] state values/counters from a
running DTester,
using these values I may implement an ok step in order to validate the range
of the values.

I don't want to give DTest complex interface but I want a "pluggable" tracer
architecture with the default one which may be TAP.

>  > Independently of the "output format" each DTester should (generically)
>  > elaborate a test result. If an ok step has failed then DTester has failed
>  > then DTestmaster know it then Campaign Manager too.
>  >
>  > > In that case, the output needs to be
>  > >   serialized it if several clients are running at the same time.
>  >
>  > Unless I miss your point I think DTestmaster already serialize
>  > the "distributed test" output?
>  >
>  > In fact there is no "real" serialize need since DTestmaster and
>  > all DTester runs on the same machine and are LOCALLY synchronized.
>  > However DTester have asynchronous "output" from the SSH Session Handler.
>
> The case I was thinking of is as follows:
>
>  D Campaign Manager -> DTestMaster -> DTest sibling
>    Understands TAP <- TAP         <- TAP
>  So it would make sense that the Campaign Manager sees:
>  TAP from client 1
>  TAP from client 1
>  TAP from client 1
>  TAP from client 2
>  TAP from client 2
>  TAP from client 2
>  Eventhough the lines where not received in that order by the
>  DTestMaster.

I think there is a misunderstanding on how current TAP output
is generated.
The TAP ouptut is generated by several OK steps of the differents
DTesters instance involved in the DTest sequence.

The TAP outputs is "automagically weaved" by the DTestMaster
itself. The "ok" method of a DTester is calling the DTestMaster one

def ok(self, *args, **kwargs):
        """Ok TAP (Test Anything Protocol) method"""
        self.dtestmaster.ok(self, *args, **kwargs)

So DTester do not produce TAP output they produce _part of_ TAP
output. For exemple when the TAP plan in announced
1..N
at the beginning of the sequence
the N is computed using the sum of all ok steps from all DTesters
registered into the DTestMaster.

look at the DTestMaster::run method code.

So the Campaign Manager will see what the DTestMaster produce,
and currently the DTestMaster "serialize" the TAP output.

>  > The tests sequence itself may/should contains expect/barrier steps
>  > which ensure synchronization when (functionnalmy) needed.
>
> Sure, my point is that the output needs to be serialized so that it's
>  easy to parse the TAP at the other end.

I think this is alrealdy done.
Moreover if your sequence contains appropriately interleaved barrier/ok steps
the output of a successful DTest sequence should be identical.
There may be be "ok step" output switch if they are not separated by barrier.


>  > I will be silent for a week, since I may not touch a keyboard during
>  > this time :=)
>
> Have a nice time then!

It was nice.
I'm back online :=)
-- 
Erk