[Monotone-devel] fun with concurrency

[Nathaniel Smith]

prompted by Joel's error message about the database schema changing, I spent a 
bit of time looking into what's up with multiple monotone's concurrently 
accessing the same database.  we have some problems.

basically, the situation is, it's not safe for multiple monotones to access 
the same database.  we're just not careful about that internally, and there 
are really good reasons to think it won't work -- for instance, there's 
monotone-version dependent state stored in the database, both in the db schema 
and in the transient VIEWs that are dropped and recreated on every execution.  
sqlite is clever and keeps track of changes in the database schema; if you 
ever let a second monotone run, it will rearrange the schema (_probably_ 
leaving it the same as it was before, but maybe not -- the VIEW layout changed 
recently, for instance, and bad things could happen if you had two different 
versions of monotone running on the same database).  rearranging the schema 
will increment a magic cookie that sqlite stores.  now the next time the first 
monotone goes to touch the database, sqlite will double-check that cookie to 
make sure the schema hasn't changed, and crash with Joel's error message if it 
has.  Moral: it is never ever okay to have multiple monotone's access the same 
database at the same time.  (Even if we changed the VIEW stuff, I still 
wouldn't be confident in it being okay without a very careful audit and 
probably some redesigns... let's not go there right now.)

fortunately, this usually isn't _much_ of a problem, because most monotone 
operations create wrap a transaction around their whole run, and end up 
locking the database the whole time; other monotone's trying to access it spit 
out:

  monotone: fatal: std::runtime_error: failure extracting schema from 
sqlite_master

and exit without doing anything.

what's unfortunate is, there _is_ a place where the lock isn't held, and two 
monotone's can run at once -- when you run 'monotone serve', it wraps each 
connection in a transaction, but when sitting idle with no connections to 
service, the server doesn't have any transaction open, and the database is 
unlocked.  fortunately this doesn't risk db corruption; if another monotone 
accesses the database during this period, it just means that the server will 
crash immediately the next time someone connects to it, leaving the database 
intact.  it's still not very friendly, though.

Conclusion: monotone needs to always keep the database locked explicitly, in 
some way that persists even over transaction boundaries.

Bad part: there is no such mechanism in sqlite, at least as far as I can see.  
maybe we should whine at upstream/submit a patch/both until they add one?  we 
could in theory create our own locking around the database, but that's a 
tremendous pain, and sqlite already has all the right locking stuff, just not 
exposed correctly for us to use.

-- Nathaniel