Re: [Monotone-devel] [sqlite] disk locality (and delta storage)

[drh]

Daniel Carosone <address@hidden> wrote:
> 
> It seems a little odd to me to build a centralised, online
> information system for tracking state and documenting activity around
> and about source code in a distributed and disconnected VCS.  
> 

Ah yes, you're right.  But in the system I envision, the wiki
and bug-tracking are also decentralized, disconnected, and
distributed.

> 
> What I'd really like to see is something that, instead of just
> plugging into monotone to show source code state and patches, actually
> used monotone for all storage and 'information transport', and allowed
> developers to update the wiki pages and bug tracking information in
> the same way they can update the code: offline, with syncs and merges
> later as needed.  

This is more in line with my thinking.

> 
> Wiki pages doesn't seem so hard, they're pretty much text documents
> stored in a VCS anyway. 

There are some complications.  Each wiki page acts if it where
its own independent project or branch.  And then you probably want
some way see all current leaves and to do merges from the web
interface.  

If you intend your system to be publically accessible, then
you will also need some mechanism to delete (or at least 
permanently hide from public view) the spam that miscreants
occasionally post on wiki pages and sometimes also on tickets.
Some kind of "death cert" perhaps.

> Bug tracking state would require a little
> more smarts, with suitable text formats stored in the VCS and
> code/hooks to assist in making sane merges between multiple heads of a
> given ticket, but even that doesn't seem terribly hard.

My thinking about tickets is that each change to a ticket
is just a small, signed, immutable record (like a cert)
that contains a variable number of name/value pairs.  The
names are things like "title", "description", "status",
"assigned-to", etc.  To reconstruct the current state of
a ticket, you sort all the pieces by their creation time
(their actual creation time, not the time they happened to
arrive at your database) and read through them one by one.
Values overwrite prior values with the same name.  So
in the end, you have one value for each field name - and
that is the current state of your ticket.

This approach gives you automatic merging and a complete
change history/audit trail.  Some people are initially shocked
that any user can update any field of the ticket, but that
kind of openness is in keeping with the wiki tradition and
has actually been used very successfully in CVSTrac.  If
you wanted to restrict changes on selected fields, you could
just ignore the name/value pairs for that field on certs
from unauthorized users.

Tickets also benefit from having "remarks" that people can
append to the ticket (without overwriting) and attachments.
Both are handled by separate certs. 

It is also very useful to have certs that record a link 
between a revision and a ticket.  In this way you can record 
that a bug was introduced by, or fixed by, a particular 
revision.  The linkage between tickets and revisions has 
proven to be particularly valuable in CVSTrac.

Once you have cross linking between revisions and tickets,
if you add a google-like search you then have a very powerful
system for doing source tree archeological work.  This comes
up a lot in maintaining SQLite.  Somebody on the web reports
a problem.  I have some vague memory of fixing a similar
problem 9 months ago, but do not recall where it was or how
I fixed it.  By using the search feature of CVSTrac together
with the links between tickets and check-in comments, I can
quickly find the historical problem and pinpoint exactly when
and how the problem was fixed.  Then post a URL to the specific
ticket that described the problem and the specific revision that
fixed it.

> 
> It could still be a web ui if people find that comfortable, just one
> that developers would often run pointing the browser at a local
> server, with a commit at the end of a session, and a later sync and
> merge. Of course, you could/would always have an instance of this
> running on a public webserver in a well-known place, just like
> monotone projects typically do for their source databases: for
> convenience, rather than necessity.

My thinking exactly.  I want the ability to drop a standalone
binary into a cgi-bin on a $7/mo hosting site and have an
instant sourceforge for some small project.  It is also nice
to be able to run things out of a database file in your home
directory.  Or do both.

Some things, like ticket reports, tend to want to use a web
interface.  So how do you do that when you're riding on an
airplane or otherwise cut off from your favorite server?
Just type (for example):

   monotone httpserver &

and then point your webbrowser at 127.0.0.1.

> 
> Some of these things might eventually go into monotone itself, perhaps
> building more tools for project policy and practice assistance around
> base mechanisms such as certs and the DAG structure.  More of it could
> be done as additional programs, with libraries of hooks to install as
> appropriate to help get their stuff done. We already have the
> beginnings of several of these things, such as testresult certs, but
> they're primitives waiting for further development as projects
> discover practical and creative ways to use them.
> 
> There might be value in this integration: perhaps having a single
> commit that fixes a bug and also updates the tracking entry for that
> bug can save some of the more manual correlation and entry work that's
> otherwise necessary for good data quality, and even assist tracking
> when the fix is later propagated to other branches.  Perhaps storing
> some of this state directly on the revisions, as certs in a structured
> format the tracking tools can interpret, will have similar benefits.
> 

I perceive that your thinking evolved as you composed your email.
At the beginning you sounded skeptical.  But here toward the end
you seem to be saying "hey, there might be some good ideas here".
I hope that you will continue to consider these ideas.  I do not
in my wildest dreams think that I have all of the answers, but
I hope that perhaps some of my thinking is sound and can lead to
better ways of doing project management.  I'm looking forward to
seeing some great enhancements along this lines in monotone in
the future!

--
D. Richard Hipp   <address@hidden>