Re: [Monotone-devel] ideas to speed up monotone SQLite database access

[Joe Wilson]

--- Matt Johnston <address@hidden> wrote:
> Assuming the OS has sufficient memory available and sensible
> buffering, shouldn't it be caching "recently-written" db
> file portions already?  We're currently committing a
> transaction after every N bytes or M packets transferred,
> which seems to be a good tradeoff between getting output
> written to disk but avoiding excessive disk latency for
> sync()s etc. Most other disk-intensive operations are
> already wrapped in transactions, there may be a few that
> still require attention.

After playing with monotone as a client, I agree with you.
Putting the database into RAM would not improve throughput
significantly. I don't believe that SQLite is a bottleneck
for the monotone client. I'd be surprised if SQLite database
time accounted for even 10% of the total time (elapsed or CPU).

The monotone client in the initial pull was idle for minutes
at a time waiting for netsync (no I/O, no CPU).  When the 
netsync packets did arrive, the CPU hovered between 80% and 
99% for a minute or two and then resumed being idle again 
waiting on the socket.

What specific monotone operation do you feel is significantly 
slowed by the SQLite database?

In order to keep the CPU sustained at 100% you might consider
a multithreading and/or multiprocess strategy to deal
with the latency of the database and socket communication. 
If you're single threaded, then the latency really gets you.
I'm guessing you should be able to schedule some CPU
intensive compression, decryption or merge operation on one 
thread while another thread is writing to the database and 
yet another thread is waiting for netsync instructions.
Perhaps the solution might be as simple as spawning a few
concurrent monotone clients that operate on the same database
doing non-overlapping work (operating on files beginning with
a different letter, etc). That way you keep the CPU pipeline
churning in one process while another might be waiting on a
socket or a database. Works for grid computations, why not
monotone?

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com