Re: [BUG/Improvement?] Large file handling.

[Larry Jones]

Eric Dumas writes:
> 
> I would like to raise an issue with cvs, when handling directories
> containing large files. We are using cvs for not only code source,
> but for data, intranet, etc. The issue is that when you do work
> remotly (US/Europe/East-Europe/Asia, etc) with one server in the US,
> a cvs update on a directory containing around 1.5 Gb of data

As you probably know, CVS was designed to handle text files, not
arbitrary data.  What benefit do you get from having such data in CVS?
You now know the cost -- is it worth it?

>     1- is taking more than 24h.

Why?  Is it simply the amount of data and the speed of the network
connection?  If so, have you enabled compression (the global -z option)?
If not, then you need to determine exactly what the performance problem
is -- perhaps the server needs more memory, a faster CPU, faster disks,
etc.

>     2- Needs a ***huge*** amount of memory on the server (size of
>     the data contained in the directory.

This fact is well known and noted in the manual.  Although it isn't
mentioned in the manual, CVS has implemented flow control for some time
now, so it shouldn't grow to the size of the data contained in the
directory but only to the size of the largest file plus the high water
mark (2MB by default).  Since your problem seems to be with a single
large file, it doesn't help in this case, but there are very good
reasons for not doing flow control in the middle of a file.

Something that's been discussed in the past that would probably help is
using the rsync algorithm (<http://samba.anu.edu.au/rsync/>) for the
client/server file transfers.  Unfortunately, no one has ever seriously
investigated doing it, so far as I know.

-Larry Jones

We seem to be out of gun powder. -- Calvin