New file system based store

[Ludovic Courtès]

Hello there!

There’s now a new file system based block store, called ‘fs’.  Currently
it has write performance comparable to GDBM (thus worse than TDB) but is
the most storage-efficient of all three methods:

--8<---------------cut here---------------start------------->8---
$ time chop-archiver -S gdbm_block_store -f ,,foo.db.gdbm --archive 
big_buck_bunny_480p_stereo.ogg 
tree_indexer:hash_block_fetcher:hash_index_handle:64::r7ua77kqdofwc3ju535zvtf4iv3vspa5/96

real    0m24.463s
user    0m1.808s
sys     0m2.247s

$ time chop-archiver -S tdb_block_store -f ,,foo.db.tdb --archive 
big_buck_bunny_480p_stereo.ogg 
tree_indexer:hash_block_fetcher:hash_index_handle:64::r7ua77kqdofwc3ju535zvtf4iv3vspa5/96

real    0m5.727s
user    0m4.064s
sys     0m1.583s

$ time chop-archiver -S fs_block_store -f ,,foo.db.fs --archive 
big_buck_bunny_480p_stereo.ogg 
tree_indexer:hash_block_fetcher:hash_index_handle:64::r7ua77kqdofwc3ju535zvtf4iv3vspa5/96

real    0m22.852s
user    0m1.964s
sys     0m3.468s

$ du -msc ,,foo.db*
165     ,,foo.db.fs
321     ,,foo.db.gdbm
253     ,,foo.db.tdb
739     total
--8<---------------cut here---------------end--------------->8---

Read performance is slightly better than the others:

--8<---------------cut here---------------start------------->8---
$ time chop-archiver -S gdbm_block_store -f ,,foo.db.gdbm --restore 
tree_indexer:hash_block_fetcher:hash_index_handle:64::r7ua77kqdofwc3ju535zvtf4iv3vspa5/96
  |sha1sum
91957ace7ff5ff71d72098a220d6a3c406d93641  -

real    0m1.860s
user    0m1.940s
sys     0m0.599s

$ time chop-archiver -S tdb_block_store -f ,,foo.db.tdb --restore 
tree_indexer:hash_block_fetcher:hash_index_handle:64::r7ua77kqdofwc3ju535zvtf4iv3vspa5/96
  |sha1sum
91957ace7ff5ff71d72098a220d6a3c406d93641  -

real    0m2.462s
user    0m3.363s
sys     0m0.424s

$ time chop-archiver -S fs_block_store -f ,,foo.db.fs --restore 
tree_indexer:hash_block_fetcher:hash_index_handle:64::r7ua77kqdofwc3ju535zvtf4iv3vspa5/96
  |sha1sum
91957ace7ff5ff71d72098a220d6a3c406d93641  -

real    0m1.695s
user    0m1.765s
sys     0m1.062s
--8<---------------cut here---------------end--------------->8---

The main advantage is its file system interface.  It makes it easy to
rsync it to a remote site, for instance, which can be useful when the
remote site cannot run ‘chop-block-server’ or similar.

Thanks,
Ludo’.

pgpKT4n67NvSL.pgp
Description: PGP signature