Lost commit bug and a fix

[Rahul Bhargava]

We have run into an issue with cvs server (we have tried 1.11.17 - 
1.12.12, all versions suffer from this)
whereby a "committed" transactions can be lost from disk due to a power 
outage or an unclean shutdown
of the machine. The OS we have experienced this were - Linux kernel 
2.4.21-27 (RHEL3) and 2.6.5, 2.6.9,
but the problem can occur on other OS as well.

When cvs commits a change to the RCS files, it invokes 
filesubr.c:rename_file() which in turns
invokes STDIO rename() function. Prior to invoking rename(), the 
rcs.c:rcs_internal_unlockfile()
invokes fclose() on the ,<file>, file which only ensures changes from 
user space buffers provided
by  the  C  library are flushed to kernel buffers. It does not flush 
kernel buffers to disk. The current
cvs code does not invoke fsync() on the ,<file>, file descriptor. 
Invoking rename(",<file>," , "<file>,v")
on Linux and almost all UNIXs only flushes the inode for the target file 
to disk, it does not guarantee
flush of the kernel buffers allocated for the ,<file>. Depending upon 
the load on the machine, the Linux
kernel's flush daemon process may not flush for a while. In the meantime 
the cvs transaction could
have been declared committed to the end CVS user (cvs process has 
returned the final "OK"). If the
machine crashes prior to syncing the changes to disk, the committed 
transaction can be lost.

In production environment we have seen this happen several times. The 
CVS server side process
needs to guarantee not just atomicity via rename but durability of the 
transaction, The solution is
to fsync(outfile fd) prior to rename(). 

The MTA/sendmail community seems to be aware of this issue. For example, 
we looked at the
source code of sendmail-8.13.5/sendmail/queue.c and confirmed that they 
use fsync and rename
pairs to guarantee changes to the files are written to disk prior to 
returning OK to the client. In the
database community we ensure we first write to a durable transaction log 
before declaring victory
on the given transaction that we are committing.  Without fsyncing its 
like playing Russian roulette
with the cvs commit. Most of the time it will work but as we found in 
production, every now and then
changes can be lost.

Note a journaling file system like ext3 doesn't help as without 
fsync/fdatasync calls the journal
may not record an event for data changes. Our production environments 
were all using ext3/Linux
when they ran into the lost commit problem.

The fix to rcs.c file :

[ccvs/src]$ cvs diff -w -rcvs1-12-12  rcs.c
Index: rcs.c
===================================================================
RCS file: /cvs/ccvs/src/rcs.c,v
retrieving revision 1.345
diff -w -r1.345 rcs.c
8450a8451,8458
>     /* Rahul: start fix , fsync the file to disk else rename can 
cause data loss */
>     if (fflush(fp) != 0)
>       error (1, errno, "error flushing file %s to kernel buffers", 
rcs_lockfile);   
>     /* Now that we have xfered to kernel buffers, lets call fsync to 
get to disk */
>     if (fsync(rcs_lockfd) < 0)
>       error (1, errno, "error fsyncing file %s", rcs_lockfile);   
>     /* Rahul: end fix
>

--
Rahul Bhargava,
CTO, WANdisco
(650) 242-8352
Mountain View, CA
http://www.wandisco.com