[Monotone-devel] SQL logic error, SQLite restore problems, etc.

[Ralf S. Engelschall]

We are currently investigating on using Monotone for use in the OpenPKG
environment. For this I imported the 26000 commits from the CVS
repository of the packaging sources (56MB raw *,v files) of the OpenPKG
CURRENT distribution via Tailor into a Monotone repository (130MB SQLite
database). The 26000 commit transactions needed about 1.5 days to
migrate via Tailor, but the result looks perfect. So far, very good.

Then we played around with the resulting Monotone repository and
stumbled over two nasty issues:

1. copying repository via raw SQLite dump/restore fails
   ----------------------------------------------------

   This is actually an SQLite problem (and I will forward to the SQLite
   community, too), nevertheless you should be aware here, too. A
   simple...

   | $ sqlite3 openpkg-import.db .dump | sqlite3 openpkg-copy.db

   ...fails with lots of errors like:

   | SQL error near line NNN: String or BLOB exceeded size limit

   The problem seems to be a too small input buffer in the sqlite3(1)
   command line tool. Ok, nothing we can do here for Monotone, but at
   least nasty as I often backup SQLite database dumps and it is nasty
   if one cannot read them in again (easily). BTW, this occurs for
   mostly all Monotone databases I tried...

2. copying repository via Monotone "pull" fails
   --------------------------------------------

   As I don't want to work with the Tailor-maintained "import" Monotone
   repository directly, I want to pull a branch from it into the actual
   "work" Monotone repository. So, I did...

   | $ rm openpkg-work.db
   | $ mtn --db=openpkg-work.db db init
   | $ mtn --db=openpkg-work.db pull \
   |   file:/v/openpkg/mtn/openpkg-import.db \
   |   'openpkg.src.cvs.current'
   | mtn: setting default server to file:/v/openpkg/mtn/openpkg-import.db
   | mtn: setting default branch include pattern to 'openpkg.src.cvs.current'
   | mtn: setting default branch exclude pattern to ''
   | mtn: doing anonymous pull; use -kKEYNAME if you need authentication
   | mtn: connecting to file:/v/openpkg/mtn/openpkg-import.db
   | mtn: finding items to synchronize:
   | mtn: bytes in | bytes out | certs in | revs in
   | mtn:   14.3 k |       174 |        0 |       0
   | mtn: bytes in | bytes out |      certs in |     revs in
   | mtn:   47.0 M |       174 |  87960/107704 | 21990/26926a
   | mtn: error: sqlite error: SQL logic error or missing database

   Bang!! Then I repeated...

   | $ mtn --db=openpkg-work.db pull \
   |   file:/v/openpkg/mtn/openpkg-import.db \
   |   'openpkg.src.cvs.current'
   | mtn: setting default server to file:/v/openpkg/mtn/openpkg-import.db
   | mtn: setting default branch include pattern to 'openpkg.src.cvs.current'
   | mtn: setting default branch exclude pattern to ''
   | mtn: doing anonymous pull; use -kKEYNAME if you need authentication
   | mtn: connecting to file:/v/openpkg/mtn/openpkg-import.db
   | mtn: finding items to synchronize:
   | mtn: certificates | keys | revisions
   | mtn:        80778 |    1 |     26926
   | mtn: successful exchange with file:/v/openpkg/mtn/mtn.db
   | mtn: bytes in | bytes out | certs in | revs in
   | mtn:      833 |       864 |      0/0 |     0/0

   ...and no more revisions are pulled, although I expected that some
   were missing. But this is not the case, the pull actually seemed to
   have worked:

   | $ mtn --db=openpkg-import.db checkout --branch openpkg.src.cvs.current 
src.import
   | $ mtn --db=openpkg-work.db   checkout --branch openpkg.src.cvs.current 
src.work
   | diff -ru3 src.import/_MTN/options src.work/_MTN/options
   | --- src.import/_MTN/options    2007-09-03 17:29:21 +0200
   | +++ src.work/_MTN/options   2007-09-03 18:03:38 +0200
   | @@ -1,3 +1,3 @@
   | -database "/v/openpkg/mtn/openpkg-import.db"
   | +database "/v/openpkg/mtn/openpkg-work.db"
   |    branch "openpkg.src.cvs.current"
   |    keydir "/u/rse/.monotone/keys"

   The problem seems to be also NOT deterministically repeatable,
   because when I retried it failed at a different revision during
   "pull" and on the third retry from scratch it even worked without the
   error. So, is this a timing problem or something like this?

You can imagine that these two issues not very much increased the
confidence of my co-workers that it is reasonable to investigate at this
time on using Monotone for OpenPKG ;-)

Anyway, the SQLite issue I'll address at SQLite.org. But the Monotone
"SQL logic error" issue and non-deterministical break on the "pull"
operation is really nasty and should be figured out (perhaps it is a
Monotone bug, a SQLite bug or even again a GCC bug).

BTW, here is the relevant version information (it mainly is the 0.36
release version of Monotone, built with Boost 1.34.1 and GCC 4.2.1 under
latest FreeBSD 6.2-STABLE):

| $ uname -a
| FreeBSD en1.engelschall.com 6.2-STABLE FreeBSD 6.2-STABLE #0: Thu Aug 30 
11:31:47 CEST 2007
| $ mtn --version
| monotone 0.36 (base revision: e4bc808d89e029ce623f9e8f2b10c84006b83fb5)
| $ openpkg rpm -q monotone boost gcc
| monotone-0.36-20070808
| boost-1.34.1-20070725
| gcc-4.2.1-20070727

PS: BTW, SQLite 3.5.0 was released today.
    Perhaps it makes a difference if Monotone is upgraded this version...

Yours,
                                       Ralf S. Engelschall
                                       address@hidden
                                       www.engelschall.com