[Monotone-devel] Re: silly question #2: why base64?

[Lapo Luchini]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Christof Petig wrote:
> The reason is that monotone was designed with sqlite2. To support binary
> data you have to use descriptors with sqlite3. The port to sqlite3 was
> done in serveral steps, first was a minimal port, second was using
> descriptors and step 3 would have been to save data in binary.

But even in sqlite the sqlite_encode function would have been more
efficient... oh, well, just another small change that *may* accumulate
in my local server ;-)

BTW: a couple fo weeks ago I began the italian translation, it's still
quite partial (but then again, also the 'fr' one still is)... would you
like to merge it already or wait a bit more?

My public monotone server may be reaced on cyberx.lapo.it and should
serve *, I guess a pull should be possibile, in anonymous mode.

Also ViewMTN on this address: http://cyberx.lapo.it/viewmtn/
(unfortunately often seems to hang and leave a stray child around...)

> Problem 1 is that binary data is ugly to handle on screen or debugging
> output.

On the other hand a diff on a "normal" text file is way more elegant
seen in text than in base64 (even if I guess that xdelta diffs have got
a bit of binary data in them anyway, at least some pieces would be
understandable text, wouldn't they?).

> Problem 2 is a transparent migration scheme for the user (that's
> not trivial).

As a first stop-gap we could modify base64encode not to encode and
base64decode to accept both encoded and non-encoded, on the assumption
that non encoded data will have at least SOME non base64 data? but this
assumption is a bit ugly and would complicate things to manage data in
BOTH formats at once... unless we decide to put a non-base-64 char as a
first char, to help specifically the decoder decide if the record is new
or old. This way even a single DB could be re-encoded step-by-step.

But probably is much better to switch at once and offer the user to
migrate it using a local "serve" + "sync" in a new DB and that's all..

> Problem 3 is to decide which data should get stored binary
> (IMHO all base64 encoded, to store IDs in binary is debatable - using
> sqlite3 as a debugging aid is not that uncommon).

Well, the base64 of the "file content" data is certain to take up the
most part of the entire DB, the content of the hexadecimal hash much
less, even if it would halve (being hex and not base64), but in case of
switch it MAY be better to switch all at once?

To debug as long as one defines an user function which does binhex or
base64encode, nothing would change at all.
Unfortunately the command-line sqlite3 command doesn't seem to support
UDFs. We could, in case, compile a cutom one directly with monotone.

BTW: it's a shame sqlite3 supports INTEGER fields only up to 8 bytes, a
big-integer-like field would have been very handy to use hashes in
numeric form and have them transparently in binary in the DB.

> Problem 4 are programs
> like monotone-viz which directly access the database ...

But I guess they wouldn't mind so much to AVOID calling a decode
function..? Shouldn't be a very difficult change, I guess/hope.

> I cannot point to the documentation, but AFAIK sqlite3 is fully BLOB
> capable (binary large object).

It seems to me that they removed all sqlite2-related "problems" form the
FAQ and the manual altogether, probably with the intent not to instill
doubts in "new" users that never encountered the limitations of sqlite2 =)

- --
L a p o   L u c h i n i
l a p o @ l a p o . i t
w w w . l a p o . i t /
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (Cygwin)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iEYEARECAAYFAkMyZ2gACgkQaJiCLMjyUvtAdQCgzgCYCmfpnUAb8jXFh4/MElN1
c/kAn2Dd1JD3YkYqBSmmgOUz0FHpWviB
=juV1
-----END PGP SIGNATURE-----