[Monotone-devel] Re: results of mercurial user survey

[Graydon Hoare]

Timothy Brownawell wrote:

> Netsync (initial pull, counting both server and client) appears to be:
>    13% libz (56% compression + 22% decompression + housekeeping)
>    11% Botan::SHA_160::hash

It's important to differentiate two classes of gzip and sha1 here. There 
is a portion we do on the networking stream (compressing and hmac'ing 
packets) and a portion we do on the database.

We can give up the gzip on either without much hassle: on the protocol 
stream it'll just result in more overall transmission (which is probably 
OK since we're not near wire speed yet) and on the database it'll mean 
more database storage allocated, which might be ok on the xdeltas but 
probably is unwise for the fulltexts. Especially full rosters; they're 
pretty big.

An easy way to reduce the gzipping of protocol packets is just to raise 
constants::netcmd_minimum_bytes_to_bother_with_gzip to something large, 
say 0xffff or 0xfffff.

To remove gzipping of xdeltas isn't likely a lot more work either; I 
think someone already went through and got rid of most of the gzipping 
of intermediate in-memory forms, so most of the gzipping which remains 
happens in database.cc. It wouldn't be more than an afternoon's work to 
purge that (though it'd make the database rows incompatible; you'd need 
a trivial schema change).

I was somewhat hesitant about hmac'ing the packets at first, and I'd 
also be willing to remove or degrade some of this if it's a major 
performance problem. We have end-to-end integrity and authentication 
info on the transferred data anyways, so it's sort of a minor win; I 
think the main thing it prevents is a possible epoch poisoning on 
initial contact with branch by a man-in-the-middle, which is pretty 
minor and easily detected and recovered from.

>    6% compute_delta_insns (strangely, this is 99% from compute_delta...
> which gets .09% of total)

Yeah, one area worth considering is an explicit xdelta inverter. It can 
be done, sorta, though it produces less-optimal deltas, and needs one of 
the input fulltexts anyways. It's still probably a win.

>    5% _int_malloc
>    5% memcpy

It's good to separate these by call context too. I'm guessing a lot of 
the memcpy is xdelta, but best not to speculate.

I'll take a stab at a few of these today and tomorrow, see what sort of 
improvement I can squeeze out.

-graydon