Re: [Monotone-devel] Testresult

[Timothy Brownawell]

On 1/5/2010 5:25 PM, Judson Lester wrote:
> On Sat, Dec 26, 2009 at 2:30 PM, Timothy Brownawell <address@hidden
> <mailto:address@hidden>> wrote:
>
>     On 11/24/2009 1:48 PM, Judson Lester wrote:
>
>         Following the bisect thread brought some questions about
>         testresult back
>         to mind.  This was one of the features of monotone that drew me
>         to it,
>         back around 0.14, and I've never really gotten around to really
>         implementing testresult in my code projects.
>
>         Is there a best practice on using the mtn testresult command?
>
>
>     No. In fact I'm not sure that it's even used much, probably related
>     to what you mentioned below. I think I know a better (more flexible)
>     approach, but it would need new hooks to be added.
>
> <snipped>
>
>     What would probably work would be a separate branch containing
>     (only?) a file with the list of (test-id, pass/fail) pairs;
>     revisions in this branch could be given an extra cert (because certs
>     are indexable, and this is something you'd use for lookups)
>     indicating which revision the contents refer to, and could maybe
>     have an extra file with information on what version of the testsuite
>     was used (for external testsuites).
>
>     Doing it this way would require a new "accept_update(old_revision,
>     new_revision)" hook, and helper functions to allow that hook to look
>     up particular revisions (use 'automate select') and then read the
>     contents of those revisions (use 'automate get_file_of'). (We
>     already have an 'mtn_automate' function that would allow these,
>     except that it will E() on its reentrancy check if called this way.)
>
>     Does this seem like a reasonable approach?
>
>
> Re-reading this, I started wondering if this might not be related to
> policy branches.  I am certainly going to reveal my novice-level
> understanding of deep version control, and possibly a complete
> misunderstanding of policy branches.  First of all, I know this was a
> hot topic in monotone about 3 years ago, but did they go anywhere?

I have a branch I've been working on on-and-off for maybe a year, 
currently I need to finish updating it after the keys-by-hash changes.

> It took me a little while to grasp the necessity of a second branch,
> although it's obvious on reflection.  In order to generate the
> testresults, you need to have the actual revision - you could only do
> that after the user issues a commit and before it's committed, and even
> then, that's a sketchy proposition.
>
> So it makes sense, then, to put the results in a cert that can be added
> later - an actor can check out the revision, run tests and then sign
> certs on the revision.  Except for the complexity of the per-testresult,
> and the data bloat you alluded to.
>
> What I started to think about then, was to commit the testresults in a
> clean format (maybe something parse_basic_io ready) into a branchless
> revision, and then sign a cert on the revision being tested with the
> rev_id of the testresult revision as the value.

I'd been thinking to have the cert be on the testresult revision, and 
point to the revision that was tested. Doing it the other way gives an 
extra benefit that even if you don't pull the testresult revisions you 
know that they're there, nice. :)

> It seems like you could build a selector for the "associated branch"
> pretty easily, and be able to check it out and examine the contents of
> the testresult.
>
> I think there'd be a hurdle with netsync, since I think (but don't know)
> if I'm syncing with a pattern, revisions that don't match the pattern
> would be excluded - so the associated revisions wouldn't make it over
> the wire.

This is why the testresult revisions *do* need branch certs - so you can 
ask for them during netsync. ;)

> There's also questions about the DAG of the associated branch...

Hmm... I guess the simplest thing would be to have the test bot(s) just 
write out their results and commit in a loop, and not care whether there 
were any other bots. So you'd end up with a linear chain of revisions 
for each bot, and with multiple bots you'd get multiple heads. I don't 
think the DAG would matter that much, since it really is a bunch of 
independent revisions that just happen to be committed as a DAG to take 
advantage of the compression.

>  From there I begin to wonder: does that look at all like policy
> branches as they've been imagined?  All I've read is the Graydon
> proposal, that implies entirely separate branches - sort of filling in
> under (e.g.) net.venge.monotone.mtn with net, net.venge and
> net.venge.monotone with update policy.

Here, each testresult revision contains info about a specific primary 
revision. With policy branches, it's the branch (so, the current head 
revision) that contains info defining the primary branch (what 
committers to trust, etc).

Think of revisions of a branch as files in a directory... a policy 
branch is the ACL on the directory and maybe the directory name; 
testresult revisions would be extended attributes on the files (just 
like certs would be).