[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Lost commit bug and a fix

From: Rahul Bhargava
Subject: Lost commit bug and a fix
Date: Sun, 25 Sep 2005 15:19:55 -0700
User-agent: Mozilla Thunderbird 1.0.5 (Windows/20050711)

We have run into an issue with cvs server (we have tried 1.11.17 - 1.12.12, all versions suffer from this) whereby a "committed" transactions can be lost from disk due to a power outage or an unclean shutdown of the machine. The OS we have experienced this were - Linux kernel 2.4.21-27 (RHEL3) and 2.6.5, 2.6.9,
but the problem can occur on other OS as well.

When cvs commits a change to the RCS files, it invokes filesubr.c:rename_file() which in turns invokes STDIO rename() function. Prior to invoking rename(), the rcs.c:rcs_internal_unlockfile() invokes fclose() on the ,<file>, file which only ensures changes from user space buffers provided by the C library are flushed to kernel buffers. It does not flush kernel buffers to disk. The current cvs code does not invoke fsync() on the ,<file>, file descriptor. Invoking rename(",<file>," , "<file>,v") on Linux and almost all UNIXs only flushes the inode for the target file to disk, it does not guarantee flush of the kernel buffers allocated for the ,<file>. Depending upon the load on the machine, the Linux kernel's flush daemon process may not flush for a while. In the meantime the cvs transaction could have been declared committed to the end CVS user (cvs process has returned the final "OK"). If the machine crashes prior to syncing the changes to disk, the committed transaction can be lost.

In production environment we have seen this happen several times. The CVS server side process needs to guarantee not just atomicity via rename but durability of the transaction, The solution is to fsync(outfile fd) prior to rename(). The MTA/sendmail community seems to be aware of this issue. For example, we looked at the source code of sendmail-8.13.5/sendmail/queue.c and confirmed that they use fsync and rename pairs to guarantee changes to the files are written to disk prior to returning OK to the client. In the database community we ensure we first write to a durable transaction log before declaring victory on the given transaction that we are committing. Without fsyncing its like playing Russian roulette with the cvs commit. Most of the time it will work but as we found in production, every now and then
changes can be lost.

Note a journaling file system like ext3 doesn't help as without fsync/fdatasync calls the journal may not record an event for data changes. Our production environments were all using ext3/Linux
when they ran into the lost commit problem.

The fix to rcs.c file :

[ccvs/src]$ cvs diff -w -rcvs1-12-12  rcs.c
Index: rcs.c
RCS file: /cvs/ccvs/src/rcs.c,v
retrieving revision 1.345
diff -w -r1.345 rcs.c
> /* Rahul: start fix , fsync the file to disk else rename can cause data loss */
>     if (fflush(fp) != 0)
> error (1, errno, "error flushing file %s to kernel buffers", rcs_lockfile); > /* Now that we have xfered to kernel buffers, lets call fsync to get to disk */
>     if (fsync(rcs_lockfd) < 0)
> error (1, errno, "error fsyncing file %s", rcs_lockfile); > /* Rahul: end fix

Rahul Bhargava,
CTO, WANdisco
(650) 242-8352
Mountain View, CA

reply via email to

[Prev in Thread] Current Thread [Next in Thread]