Re: [Monotone-devel] Performance improvement splitout

[Eric Anderson]

Nathaniel Smith writes:
 > On Mon, Sep 04, 2006 at 05:42:45PM -0700, Eric Anderson wrote:
 > 
 > > net.venge.monotone.experiment.performance.vcache-size-hook
 > 
 > [ redo after recent roster changes ]

Ok.

 > > net.venge.monotone.performance.experiment.botan-gzip
 > 
 > My memory on this one is that I didn't see anything wrong with not
 > clearing gzip memory, but I wasn't qualified to review the
 > implementation?

I think I may have just remembered the "didn't see anything wrong with
it bit".

 > [ printf("XXX\n"); ]

that's debugging code, weird that I missed it, I'll fix it subject to
the next discussion.

 > but still), the comment on "paranoid_memory_clearing" is misleading
 > (it seems to say that if set, Botan in general will not memset(0)
 > things, but if you've ever tried this, Botan crashes horribly,
 > paranoia isn't really the issue), and I don't think the API is one we
 > could convince Jack to take upstream.

Yes, I did try that because botan was still consuming a big chunk of
time in memset, but as you said it doesn't work.  Suggestions on how
to make the comment clearer?

 > The idea is good, but I think we can manage this a little more
 > elegantly without a big investment.

Ok, do you have ideas as to how?

 > > In addition to the ones that have been discussed earlier, in testing
 > > with pulls over the local network, I found that both sides would idle
 > > during the transfer.  Interestingly, this doesn't happen in testing on
 > > one machine, even with a dual CPU machine so that both server and
 > > client can run at the same time.  I haven't split this one out yet:
 > > 
 > > 5cc1ed0346c5129ddd77ae895985bf743f349cae: skip calling select
 > > Skip calling select again if we processed some data from the remote
 > > side.  Somehow the call to select with a 1us timeout ends up waiting a
 > > bunch of the time leading to idleness in the client (on a pull).
 > > Oddly making this change significantly increases the amount of user
 > > time; haven't investigated why as the tradeoff for reduced wall clock
 > > time is a win.  Best guess is that because the client is running
 > > faster, it makes more recv calls for less data.
 > 
 > Hmm, I don't get it -- if the socket is readable (and presumably it
 > is, if we're getting bytes when we call recv), shouldn't select be
 > returning instantly anyway?  Your description seems to indicate that
 > it's actually blocking...

it's not blocking, it's just waiting an extended period of time.  If
you run strace in the -T mode to print time spent in syscalls, I
discovered that even when calling with a timeout of 1us, it would
still wait for a few milliseconds.  It didn't happen all of the time,
and the call was only checking for error conditions, so whether or not
data was present to be read was irrelevant.  

Stuff like this:
% strace -e select -T mtn -d /tmp/foo.db pull usi.hpl.hp.com \*     
select(8, [], [], [7], {0, 1})          = 0 (Timeout) <0.004812>
select(8, [], [], [7], {0, 1})          = 0 (Timeout) <0.004714>
select(8, [], [], [7], {0, 1})          = 0 (Timeout) <0.005282>
...

not
select(8, [7], NULL, NULL, {21600, 0})  = 1 (in [7], left {21600, 0}) <0.000010>

>From what I recall and the patch, it was this part:
      probe.clear();
      probe.add(*(sess.str), sess.which_events());
      Netxx::Probe::result_type res = probe.ready(armed ? instant : timeout);

in the armed == true state.

Waiting ~5ms on average was causing CPU utilization to hover around
80%, even though the system as a whole was capable of 100%.  As I said
earlier, for whatever reason, when talking over the loopback device on
one machine, the same behavior was not observed.
        -Eric