RE: cvs corruption during tag operation

[Jeremy Todd]

Hi all,

Thank you so much for the pointers and suggestions. We spent a while looking
at this problem, and after a few days of diagnostics (the issue took several
hours to reproduce, so it was slow going), we concluded that it was hardware
and not software related. It's tough to be sure, but we cannot reproduce the
issue at all with a new motherboard and hard drive but the same exact
partitions (we just copied the BSD partitions to the new system and it
booted without problems).

I didn't think file corruption like this (a single bit in a file flipping,
without any warning) could occur with modern hardware. I knew that RAID, ECC
RAM, etc can help, but I thought that regardless of hardware, the worst case
was detected corruption not undetected corruption.

Anyway I know this is off-topic for the list now. I'm pretty sure the
problem is not cvs-related, so thanks again for the quick feedback and next
time I won't be so quick to blame cvs :).

Regards,
Jeremy


> -----Original Message-----
> From: Jeremy Todd [mailto:jeremy@izotope.com] 
> Sent: Thursday, February 02, 2006 11:08 AM
> To: 'bug-cvs@nongnu.org'
> Subject: cvs corruption during tag operation
> 
> Hi,
> 
> I've managed to reproduce data corruption which is apparently 
> caused by a cvs tag operation. I'm working on finding the 
> exact cause of the bug, but here's what I've done thus far:
> 
> We're running CVS client 1.11.17 from a Windows XP machine, I 
> downloaded the binary a while ago, possible from WinCVS or 
> Cygwin, but I'm not sure. Our CVS server is running FreeBSD 
> 5.4 and we connect to it via ssh (our CVSROOT looks like 
> :ext:x@y:/cvsroot).
> 
> We have a fairly large (7138 files, 657 MB) module containing 
> both text and binary files. Our build scripts check out this 
> module and tags it fairly often (perhaps 20-30 times each 
> day). We noticed data corruption in a file in this module, 
> and I was able to reproduce it with a perl script that does 
> something like this:
> 
> while(true) {
>   cvs co X
>   cd X
>   cvs tag test-n (where n is the current iteration)
>   cd ..
>   move X Y
>   cvs co X
>   diff -r X Y
> }
> 
> after about 90 iterations the diff will fail. In every case, 
> a single character in one of the files is modified by turning 
> ON the second bit. In other words the hex value of the 
> character is increased by 2.
> 
> This has happened in both text and binary files. When it 
> happened in source files, it resulted in a compilation error, 
> which is how we caught the corruption. Unfortunately it will 
> be more difficult to determine if this has happened in a 
> binary file, and in those cases I fear we'll have some 
> serious problems down the road.
> 
> Note that when the files are corrupted, it appears to be a 
> single byte in the x.y,v file in the CVS repository that's 
> corrupted. I have not yet learned how CVS is implemented, but 
> it seems that changes made directly to the files in the 
> repository can go unnoticed by CVS (no checksums?), so it's 
> difficult to know whether it's CVS, SSH, or the filesystem 
> that's causing the corruption.
> 
> Any insights on how to debug further would be most 
> appreciated. I wasn't able to find any similar known bugs (at 
> least not for cvs versions as late as 1.11.17). Unfortunately 
> it takes about 6 hours to reproduce, so it's going to be a 
> challenge to say the least.
> 
> Regards,
> Jeremy Todd
> iZotope, Inc.
>