[Monotone-devel] Benchmarking tool

[Nathaniel Smith]

Well, I've been complaining for months about the lack of any
systematic benchmarking suite, so I've finally been trying to do
somethign about it.  It's still pretty raw, but it seems to basically
do something that works.

To play with it, check out net.venge.monotone.contrib.benchmark.

Requires python 2.4.

The current README file:

---------------------------------------------
Try:

If on linux 2.6.16:
$ gcc drop_caches.c -o drop_caches
$ sudo chown root.root drop_caches
$ sudo chmod u+s drop_caches
Otherwise, enhance drop_caches to work on your platform, or do
without.

Then:
$ python2.4 benchmark.py \
   -m mtn-0.26=path/to/0.26 -m mtn-0.27=path/to/0.27 \
   -b pull='Pull(ExistingRepo("path/to/some/repo.mtn"))' \
   myscratch myresults

Then look in myresults/.


Add new instrumenters (oprofile, callgrind, massif, dtrace IO tracing,
...) to instrumenters.py.

Add implementations of the mtn.py interface backed against, say, hg
and git, in new files, to do side-by-side comparisons (use the -v
switch, which is a more general form of -m).

Add ways to generate synthetic histories to repo.py, so we can stop
screwing around with stupid arbitrary test databases that can't be
varied systematically (even in size!) or compared between people.

Add new benchmarks to benchmarks.py.  This probably will require
enhancing mtn.py.  Workspace benchmarks are needed too, and should get
an abstract interface like repo.py.

Extra points: implement some sort of caching mechanism or something,
to reduce the amount of copying of large files required.




Oh yeah -- Copyright (C) 2006, Nathaniel Smith <address@hidden>.
Licensed under the terms of the GNU GPL v2, or, at your option, any
later version.

---------------------------------------------

It currently has exactly 1 benchmark (fresh pull), 1 repo source
("use this db I already have sitting on disk here"), and 1
instrumenter (good old time(1)), but it's already useful.  I plan to
keep enhancing it, probably tackling things in reverse order of how
the previous sentence listed them; any help would also be appreciated,
if anyone else is interested.

Cheers,
-- Nathaniel

-- 
When the flush of a new-born sun fell first on Eden's green and gold,
Our father Adam sat under the Tree and scratched with a stick in the mould;
And the first rude sketch that the world had seen was joy to his mighty heart,
Till the Devil whispered behind the leaves, "It's pretty, but is it Art?"
  -- The Conundrum of the Workshops, Rudyard Kipling