Re: [Monotone-devel] basic_io inventory

[Christian Ohler]

Stephen Leake, 2007-04-24:

> Thomas Keller <address@hidden> writes:
>
> > b) Since we already have those data in place: I'd vote for including
> > the old and new fileid in the stanzas (where applicable)... this saves
> > additional calls to automate identify or automate get_manifest_of for
> > the interface programmer.
>
> I don't see many commands that take a fileid; the commands most
> interesting from an Emacs interface point of view just need the file
> path. So perhaps fileid output should be requested via an option, to
> save parsing time in the typical case. 

I'm not convinced that omitting file ids is likely to be a big 
performance win in xmtn.  This is a constant factor, probably small.

Omitting all "uninteresting" files sounds much more promising.  This 
changes the order of the number of files.

As a benchmark, I added an Emacs Lisp function 
`xmtn-tests--parse-basic-io-inventory-benchmark' to my DVC branch that 
runs mtn automate inventory and parses its basic_io output.  It just 
counts the number of changed, ignored and unknown files.

This is a meaningful benchmark because I'm interested in the amount of 
time that can be saved by making mtn omit "uninteresting" files instead 
of printing them and letting xmtn filter them out.  Just counting their 
number is a very basic way to verify that xmtn is really receiving the 
data and doing the filtering.

On my machine, mtn automate inventory (from 
65dcf7bd70c798d6a8d4216628ad664223e52295) needs about 1.13s wall time to 
generate its output for its own source tree (with hot caches, without 
inodeprints, output redirected).  (Stephen, I suspect you're printing 
the output to the terminal, and your terminal is taking >20s to display 
it all.)

The benchmark function needs about 1.80s wall time (including garbage 
collection, assertions disabled, average of 100 runs); this includes 
running the mtn subprocess.

This is on a dual-core machine; however, in this case, there isn't much 
parallelism between mtn and xmtn.  xmtn only starts parsing the output 
after mtn has finished.  (This is a limitation in xmtn's basic_io 
parser: It doesn't parse input incrementally.)

So the time that xmtn needs to filter out uninteresting files from the 
monotone tree seems to be about .67s.  The Emacs Lisp code needs 37% of 
the total time, 63% are spent in mtn.

Since this is an interactive operation, the total time of 1.80s would 
still be a noticeable delay.

One possible improvement that is easy to test is enabling inodeprints. 
This cuts down mtn's time by .17s to about .96s.  xmtn still needs the 
same amount of time (as expected), for a total of 1.63s.

Reducing the amount of data that Emacs has to parse could reduce the 
total time by up to 37%, plus the time that mtn can save by not printing it.

However, simply making xmtn unconditionally take all the inventory data 
from mtn has its appeal, too.  While the default is not to show files 
that don't "need attention" (e.g. ignored, unmodified), there will have 
to be a switch to let the user see the ignored files, too.  And having 
to run mtn again at that point to retrieve the missing data would 
complicate xmtn's logic.  Slightly, anyway.  For the first cut, the 
simpler variant may be preferable.  On the other hand, maybe the first 
cut shouldn't even include a switch to show ignored files, or worry 
about performance at all...

Stephen, if you want to try running the benchmark function yourself 
(possibly with your variant of the mtn binary that skips "uninteresting" 
files): Merge from my DVC branch, disable assertions in xmtn-base.el, 
rebuild and reload DVC, compile and load xmtn-tests.el and run M-x 
xmtn-tests--time RET (xmtn-tests--parse-basic-io-inventory-benchmark 
"/path/to/mtn/executable/from/the/basic_io/inventory/branch" 
"/path/to/mtn/workspace") RET.

Christian.