[Monotone-devel] A conversation about workflow and quality control

[Nathaniel Smith]

This is a longish conversation I had last week with some of the OE
guys in their IRC channel.  It covers a lot of stuff about workflow
management, patch review, that sort of things; people interested in
such topics might find the discussion of how to do them with monotone
interesting!

<zecke> njs: and one thing that excites me and koen about monotone
        is: *drumbeat*
<zecke> *drumbeat*
<zecke> QA support. I plan to integrate the results of the tinderbox
        into a monotone DB
<zecke> and then automatically generate reports on a website
<njs>   zecke: woo
<zecke> njs: e.g. we know package xyz version 3.2 was
        build for XScale on OpenZaurus
<zecke> njs: we want to mark it, and then extract the information
<zecke> njs: and it sound doable, right?
<njs>   zecke: sure
<zecke> and this would make us a power user of monotone ;)
<njs>   have you heard the story about how monotone started?
<njs>   graydon was working on a contract writing something; the client
        set up some wacky setup where there were two CVS repos.  One
        of them, he committed to.  Then there was a rsync script to
        pull changes out of that one, run the test suite, and only if
        the tests pass, commit them to the other repo.
<zecke> no
*       zecke gets some popcorn
<njs>   the other repo is the one the clients used.  So, if he committed
        broken stuff, they just never even saw it.  So much for "it
        compiles, ship it"...  Totally kicked his butt.
<njs>   so, like, one of the total inspirations for monotone was, making
        that kind of workflow possible :-)
<njs>   multiple heads are there so when someone else screws up, you can
        just ignore it and keep working in parallel with their
        screwups; certs are there so you can add arbitrary data about
        things like "this rev fails tests", and monotone can use those
        to make decisions...
<njs>   then, it turned out that it takes like 3+ years just to build a
        decent VCS, never mind awesome workflow stuff too :-) so the
        support for this stuff is a little undeveloped; though you
        _can_ teach 'update' to just ignore revisions that don't pass
        certain tests (the exact set of tests is a workspace setting).
<RP>    njs: Is it possible to have a revision require certs from x
        developers before it becomes visible?
<RP>    ("visible" being merged into the current head)
<njs>   RP: like I said, it's a little undeveloped; there's nothing
        built in to handle "x developers must affirm" ATM
<njs>   RP: but the basic pieces to build such a thing are there (e.g.,
        you could have an "approve" cert, and have a little bot that
        watches commit traffic and when a rev gets x "approve" certs
        signed by keys that the bot likes, it sticks a new branch cert
        on it)
<njs>   (for now, it'd be a convention that only the bot commits to the
        branch in question; when we have better permission rules, it
        would become enforced.  Better permission rules might eliminate
        the need for the bot, too, perhaps; dunno yet.)
<njs>   RP: the other important thing is that we're really interested in
        doing this sort of thing, so while I can't say "here, just do
        this, it's worked great for lots of people", instead you'd be
        experimenting with new stuff, it's new stuff that we really
        want to make work, and there's no real way to do that except if
        some projects start experimenting with it :-)
<RP>    njs: It does seem like a really attractive feature and one I can
        see OE potentially using in the future
<njs>   RP: yeah
<RP>    Its quite neat its possible at the moment with a bot :)
<njs>   RP: having structured process makes a huge difference on
        development.  _so_ much easier just to know what's going on,
        and not always feel like you're panicked, confused,
        overwhelmed...
<RP>    njs: We were considering whether we should limit access to certain
        core components of opembedded such as the classes directory. We
        disucssed the idea of having changes to those areas only made
        after its agreed (signed?) by two other people (or whatever).
<njs>   code review likewise, doesn't have to take much time at all to
        get major benefits (who writes books without a proofreader?
        and code is a lot easier to screw up than prose!)
<njs>   nod.  don't really know enough about that code or your culture to
        comment :-)
<RP>    njs: Certain bits have a lot more impact that other outlying parts
        and we can see a need to add more protection to those critical
        bits
<RP>    Recently we've given new developers access and they've not been
        entirely familiar with the addmittedly largely unwritten code
        of conduct we sort of follow :-/
<njs>   a general principle I know is that one should be wary of the
        difference between structure and control -- there's often an
        temptation to do things like give people super-fine-grained
        permissions (you can write to _these 3 dirs_ only!), etc.
<njs>   but it seems like in practice, it's better to use technical
        means as a guide and to provide structure -- help you keep
        track of what's going on, make it take explicit work to do
        something weird, etc. -- and leave enforcement itself to social
        conventions
<njs>   (e.g., http://www.producingoss.com/html-chunk/vc.html#vc-authz )
<RP>    We basically agreed we'd want this protected area as small as
        possible
<njs>   like, maybe it would be better to have a rule "commits to this
        area shoudl be read over by at least one other core contributor
        before being integrated", than "this area cannot be modified
        unless you hunt down some other people and get them to stamp
        off on it"
<njs>   (again, dunno what you were actually talking about, just some
        general experience
<njs>   )
<RP>    njs: We were also talking about the idea of asking for patches
        touching these areas to be run past the mailing list first
<RP>    njs which should be the general princpiple anyway...
<njs>   RP: nod.
<njs>   RP: that's a common trick.  It'd be nice if there was more tool
        support for that sort of thing, e.g. in the VCS 
<njs>   RP: though I admit I'm not sure what exactly the tools should do
        :-)
<RP>    njs: The idea of signing off the change and only activating the
        patch after two other sigs was one way a tool might handle it
<njs>   yeah
<RP>    njs: Part of the problem is that OE doesn't exactly have a leader
        at the moment so people knowing what they can and can't touch
        is tricky :-/
<njs>   remember cradle-to-grave, though -- need some way to notify
        people when new patches are posted, what happens to patches that
        needed to be sent back for more changes, how do I find patches
        to review.. :-)
<njs>   nod.  though leaders are kind of overrated in FOSS anyway, it's
        not like they actually can _lead_ things very much :-)
<RP>    njs: Maybe we need a branch for this patch tracking :)
<RP>    and mutliple heads would tell you how many patches needed
        reviewing :}
<njs>   that's one option :-)
<njs>   you might get a little more mileage out of the tool by saying
        one patch=one branch
<RP>    I'd be happy with that, if we could group braches
<RP>    Obviously some branches are going to have much greater importance
        than others and we're need a way to split them into categories,
        a bit like a bug status in bugzilla
<RP>    I suspect certs might come into this somewhere but I'm no expert
        :)
<njs>   that way, it's easy for people to collaborate on a single patch,
        etc.  (the conventional review cycle is "read code, write notes
        on it, send the notes back as a list of things to fix" -- but
        this is silly, if there are obvious fixes, the reviewer might
        as well just make them if they want.  branches easily support
        multiple people working on them simultaneously...)
<njs>   http://mt.xaraya.com/com.xaraya.core/index.psp
<njs>   ^^ xaraya uses some special certs on branches to give branch
        descriptions and statuses; then they have a web tool that finds
        the most recent such certs on each branch, and displays them
        all together
<RP>    If you look at bugs.openembedded.org, we probably have 300 open
        bugs and maybe 100 open patches in there which is going to be a
        lot of branches to list without some kind of filtering
<RP>    I'll show zecke that link as he's thinking of something similar
        with his tinderbox status reporting
<njs>   it'd be possible to do something similar with statuses like
        "finished" (doesn't show up on the list by default -- or just
        infer this from whether or not its been merged to trunk!  you
        can read that out of the history graph...), "ready for review"
        (query for these to find things to review), "reviewed and ready
        for integration", etc.
<RP>    but from the commandline tool, you wouldn't get the filtering and
        such filtering would be tricky as everyone would have a
        different set of needs for statuses?
<njs>   (oh, also a useful link on conventions and permissions:
        http://www.producingoss.com/html-chunk/vc.html#vc-authz ... actually,
        that whole book is rather nice.)
<njs>   you could write a command line tool that knew how to do that
        kind of filtering :-)
<njs>   (probably it would be the right first step, lot easier to
        prototype things and try them out that way, then building them
        into monotone's core.)
<RP>    yes, I can understand that. montone would then just be the
        backend. Its got a lots of attractions to it :)
<njs>   it's not very hard to interface to mtn from, say, python;
        there's a reasonably extensive set of commands to query the db,
        and python interfaces to them. (and we're always happy to add
        more commands, if it turns out there are useful things that are
        hard to do.)
<RP>    I'm not a python guru but we have people who are :)
<njs>   yeah, I was thinking about bitbake :-) most of the automation
        commands are even easy to interface from sh, which is about as
        lowest-common-denominator as it gets...
<RP>    I have been known to work on bitbake, despite not knowning python
        very well :)
<RP>    I can manage sh :)
<RP>    I'd best get some sleep. 'night all


Cheers,
-- Nathaniel

-- 
  /* Tell the world that we're going to be the grim
   * reaper of innocent orphaned children.
   */
-- Linux kernel 2.4.5, main.c