[Monotone-devel] Bugs, features and history (was Re: pretty pretty pictures (for some value of pretty))

[Graydon Hoare]

Koen Kooi wrote:

> I still think monotone is the best open-source VCS/SCM around, but I am a bit 
> dissapointed
> at 'history digging' tools and their performance. And more recently in 'mtn 
> add' and 'mtn
> diff' being broken by the IMO bogus restriction code. So I'm wondering, is 
> there a plan to
> fix bugs like this, or is the plan to keep adding more features like policy 
> branches and
> do a bug-sprint after that?

I think it's a mistake to view policy branches as "new feature" work. 
It's new vocabulary, but it's related to a concrete cluster of bugs. I 
want to draw your attention to history for a moment.

Three and a half years ago the biggest "foreground" cluster of bugs 
looked like this:

  - The NNTP protocol can get wedged on some news servers.
  - The HTTP protocol can get wedged on some web servers.
  - Depots can get desynchronized from clients such that receive order
    is wrong.
  - Queueing packets for transmission can go wrong and send stuff that
    gets silently dropped.
  - It's not clear how to handle new depots that share a bunch of
    history. Partial re-send?

And we were miserable and grumpy and convinced the program sucked, but 
then sobered up and looked and saw the problems all centered around a 
bunch of code that's completely forgotten now. So we sat around 
discussing set-theoretic problems for a while, and bloom filters and 
merkle codes and stuff. We finally decided that on-the-fly 
synchronization was a better deal, cut out the networking and 
packet-queueing code, and replaced it with netsync.

Two and a half years ago the biggest "foreground" cluster of bugs looked 
like this:

  - The merger and other history-tracing code can get lost in history
    cycles.
  - Two projects can have their histories conjoined by accident.
  - Malicious users can time-travel.
  - File identity is being inferred in an unreliable way.
  - Renaming things produces certs, so file identity depends on
    trust hooks?

And we were miserable and grumpy and convinced the program sucked, but 
then sobered up and looked and saw the problems all centered on 
patch_set.cc (also now long dead). So we sat around discussing 
composable change algebras and immutable histories and iterated hashing 
and stuff. We finally decided that explicit, linked history logs were a 
better deal, cut out the implicit patch_set code and replaced it with 
change_sets and revisions.

A year and a half ago the biggest "foreground" cluster of bugs we had 
looked like this:

  - Merge can go crazy and produce illegal history.
  - Moving directories can cause crashes.
  - Restrictions get confused and include or exclude the wrong files.
  - Independent adds can cause irreconcilable histories that just crash
    the merger.
  - Sometimes the merger silently succeeds and trashes your work when
    it should not.

And we were miserable and grumpy and convinced the program sucked (even 
Linus said so!), but then we sobered up and looked and saw the problems 
all centered on change_set.cc and the godawful merge "algorithm" in 
there (I use this term loosely), and to a lesser extent on the continued 
use of manifest.cc. So we sat around discussing graph theory and user 
models and file identity stuff. We finally decided that a 
non-transmitted cache of file identities and cached 
quasi-graph-dominance information was a better deal, cut out the 
change_set and manifest code and replaced it with cset, roster and 
multi-mark-merge.

Now the biggest "foreground" cluster of bugs looks like this:

  - Need to rename branches
  - Need to mark branches as "done" so they stop polluting UI
  - Need to rename keys
  - Need to kick users out of projects occasionally
  - Need a way to denote "trusted core developers" and similar
    subsets of devs (translators, etc.) in a simple way
  - Need a way to migrate to ever-stronger crypto as algorithms break
  - Need a way to automatically set up new users with the trust rules
    of a project, without having them edit lua files.

A few things stand out here: first that these bugs are actually all 
*very old* bugs that have been deferred for years, second that they're 
not so much breakage as "incorrect initial implementation" so they smell 
a bit more like features, and third that they're all relatively shallow 
and shouldn't even require a history-graph rebuild once we figure out 
the right structure. That's *fantastic* news to me; it suggests we're 
actually making the set of bugs smaller over time. It almost restored my 
faith in progress!

Following the pattern, I think we've passed the misery and drinking 
stage, so I expect a flurry of discussion on theory for a while, then an 
experiment that cuts out the rotten old parts (hint: cert.cc, keys.cc, 
update.cc) and replaces them with something conservatively reflecting 
current thought and experience, followed by a release that solves a 
whole whack of problems at once. And as the ROADMAP file has indicated 
for some time, I think this might actually be one of the last big 
cut-and-replace jobs (merge-into-dir being the other, but it seems to be 
happening concurrently).

Does this mean that there's less interest in more isolated, one-off 
bugs? Possibly a bit, but no more than the historical average. 
Everyone's attracted to big-bang fixes. They're dramatic. But I don't 
think that fact represents a strategic shift away from "addressing 
bugs". That's still the primary motivation. We'll have another sprint to 
gather up the forgotten, less glamorous one-off bugs someday too.

-graydon