Re: Improving CVS Performance?

[Greg A. Woods]

[ On Saturday, March 29, 2003 at 08:34:13 (+0530), Vivek Venugopalan wrote: ]
> Subject: Improving CVS Performance?
>
>    We have a 2 GB repository running on a Linux system for over an year.  It
> has a few thousand files (approx 10,000 ) and has started progressievly
> become very slow.  We have a lot of tags (daily builds with a tag for each
> build) and a few branches ( < 10).  Can you folks suggest what are the
> things I can do to improve the overall performance of the system?

Get rid of any unnecessary tags -- e.g. something like:  remove all the
daily tags older than a week, keeping only weekly tags, then remove all
the weekly tags older than a month keeping only the monthly tags (and
arrange the rest of your SCM procedures to fit this "schedule").

> For example, a cvs update takes close to half an hour for the entire
> repository on the local box itself.

That's not too unexpected I suppose.  What you should expect depends
almost entirely on your disk I/O system and filesystem implementation
and how much RAM for buffer cache you have.  CVS is all about I/O
througput (especially when just running on the local system) and the
better your OS can cache FS data then the fewer I/Os will be needed.

Note also that Linux (like NetBSD prior to 1.6) have never been known to
be stellar performers at optimal filesystem cache utilisation.  I would
choose FreeBSD-4.x for my CVS server if I were you.

With today's blindingly fast CPUs you can "waste" a lot of cycles just
to save a few I/Os and you'll still come out way on top, and the way to
do that is get your disks as close to your host bus as possible and then
use OS-based logical volume striping to get I/Os going in parallel and
as fast as possible.  Of course you also should use mirroring to keep
things reliable enough to use in production.

A decently fast (> 300MHz) Intel Xeon CPU (i.e. with at least 2MB of
cache RAM), a GB of RAM or two, with four or six (or so, an even number)
15,000 RPM Ultra-320 LVD SCSI drives on a matching high-end Adaptec
controller (or pair of controllers on a motherboard with at least two
completely separate PCI buses), and using Vinum for software striping
and mirroring (i.e. create two identical pairs of striped volumes from
half your disks and then mirror them) should give you a reasonably fast
CVS server.

I have my /cvs on one external RAID-5 box and my /work on another
(CRD-5500's with 20 MB/s host interfaces but older 8-bit drives and not
enough cache RAM), both on the same host adapter (AIC-7880 onboard).

Here are sample times from my nightly update of my local NetBSD working
directories from a local copy of the NetBSD repository:

        START:2003/03/28-23:55:30: cvs update /work/NetBSD/src
        [[ ... about 50 files updated ... ]]
        DONE:2003/03/29-00:18:41: updating /work/NetBSD/src

        START:2003/03/29-00:18:41: cvs update /work/NetBSD/src-1.6
        [[ ... no files updated ... ]] 
        DONE:2003/03/29-00:45:40: updating /work/NetBSD/src-1.6

        START:2003/03/29-00:45:40: cvs update /work/NetBSD/xsrc
        [[ .. one file updated ... ]]
        DONE:2003/03/29-00:53:53: updating /work/NetBSD/xsrc

        START:2003/03/29-00:53:53: cvs update /work/NetBSD/pkgsrc
        [[ ... almost 200 very small files updated ... ]]
        DONE:2003/03/29-01:26:14: updating /work/NetBSD/pkgsrc

According to the nightly RSYNC job that runs just before the above
updates the repository contains:

        Number of files: 187695
        Total file size: 2193649940 bytes

(the majority of those files are in pkgsrc, but they're all quite small)

Here's another view where /ocvs is approximately (within 10%) the
non-NetBSD portion of what's all combined in /cvs now:

Filesystem  1K-blocks     Used    Avail %Cap  iUsed iAvail %iCap Mounted on
/dev/sd0f     2834732  1966837   726158  73% 183374  529072  25% /ocvs
/dev/sd5c    12186596  4428041  7149225  38% 379877 2667545  12% /cvs


The FreeBSD system configuration outlined above should be at least an
order of magnitude faster than my system.

-- 
                                                                Greg A. Woods

+1 416 218-0098;            <address@hidden>;           <address@hidden>
Planix, Inc. <address@hidden>; VE3TCP; Secrets of the Weird <address@hidden>