Re: Another dead lock in libdiskfs / ext2fs

[Thomas Schwinge]

Hello!

On Tue, Apr 10, 2007 at 01:48:17PM +0200, Thomas Schwinge wrote:
> Still while trying to get us an up-to-date git binary package I saw its
> test suite hang, system got instable, tried to attach gdb to the affected
> ext2fs instance, system crashed.  Barry restored the system.  This time I
> first attached a ``noninvasive'' gdb to the ext2fs instance, re-ran the
> test suite, saw it hang at the same place, examined the situation in gdb
> and saw that one of the ext2fs threads was doing a `dir_lookup' for git
> and was hanging there and the syncing thread was hanging as well,
> probably while trying to write the very same inode to disk?
> 
> Unfortunately I failed to move the exact gdb output to a save place
> before I managed to crash the box again.  Gah.

So, here we go.  Apply the attached patch after getting a recent git
version (from the upstread git repository, for example).  Then build it,
set `$PATH' appropriately and run the following (simplified from the
hanging test suite case):

#v+
$ git init-db
$ echo foo > foo
$ git add foo
$ git-commit -m Foo. foo
$ git-branch -l n/n
$ git-branch -m n/n n
remove_empty_directories: 
f=.git/logs/refs/heads/n;p=.git/logs/refs/heads/n;l=22 < 4096.
remove_empty_dir_recursive: p=.git/logs/refs/heads/n;l=22.
[Hangs.]
#v-

(Beware the the file system will be in an inconsistent state now, so
you've better done this on an experimenting-only file system.)

gdb on git-branch:

#v+
(gdb) bt
#0  0x011b191d in _hurd_intr_rpc_msg_in_trap () from /lib/libc.so.0.3
#1  0x01326d39 in dir_lookup () from /lib/libhurduser.so.0.3
#2  0x0119d3f8 in lookup_op.7513 () from /lib/libc.so.0.3
#3  0x0119824e in _hurd_ports_use () from /lib/libc.so.0.3
#4  0x0119d4db in hurd_file_name_lookup () from /lib/libc.so.0.3
#5  0x0119db29 in file_name_lookup () from /lib/libc.so.0.3
#6  0x01271807 in open64 () from /lib/libc.so.0.3
#7  0x012379c3 in opendir () from /lib/libc.so.0.3
#8  0x08090011 in remove_empty_dir_recursive (path=0x1015a1c 
".git/logs/refs/heads/n", len=22) at refs.c:668
#9  0x08090267 in remove_empty_directories (file=0x80fb600 
".git/logs/refs/heads/n") at refs.c:734
#10 0x080934ea in rename_ref (oldref=0x101bb98 "refs/heads/n/n", 
newref=0x101ab98 "refs/heads/n",
    logmsg=0x1016b34 "Branch: renamed refs/heads/n/n to refs/heads/n") at 
refs.c:974
#11 0x08057373 in rename_branch (oldname=0x101f00e "n/n", newname=0x101f012 
"n", force=<value optimized out>) at builtin-branch.c:519
#12 0x08057e97 in cmd_branch (argc=4, argv=0x101eda4, prefix=0x0) at 
builtin-branch.c:645
#13 0x0804a8e4 in handle_internal_command (argc=4, argv=0x101eda4, envp=<value 
optimized out>) at git.c:324
#14 0x0804b2a9 in main (argc=4, argv=0x22, envp=0x101edb8) at git.c:361
#v-

gdb on ext2fs (I think I picked the only interesting thread):

#v+
Thread 3 (thread 247.3):
#0  0x010dbacc in mach_msg_trap () from /lib/libc.so.0.3
#1  0x010dc35f in mach_msg () from /lib/libc.so.0.3
#2  0x010a7602 in cproc_block () at 
/var/tmp/hurd-io+stat+fatfs/hurd-20060825/build-tree/hurd/libthreads/cprocs.c:643
#3  0x010a7c76 in __mutex_lock_solid (ptr=0x806a564) at 
/var/tmp/hurd-io+stat+fatfs/hurd-20060825/build-tree/hurd/libthreads/cprocs.c:954
#4  0x08054513 in diskfs_cached_lookup (inum=28, npp=0x12abb98) at 
/var/tmp/hurd-io+stat+fatfs/hurd-20060825/build-tree/hurd/ext2fs/inode.c:78
#5  0x0104ad89 in diskfs_check_lookup_cache (dir=0x80698e8, name=0x12adf01 "n")
    at 
/var/tmp/hurd-io+stat+fatfs/hurd-20060825/build-tree/hurd/libdiskfs/name-cache.c:256
#6  0x0104a9b4 in diskfs_lookup (dp=0x80698e8, name=0x12adf01 "n", type=LOOKUP, 
np=0x12abcd0, ds=0x0, cred=0x8065440)
    at 
/var/tmp/hurd-io+stat+fatfs/hurd-20060825/build-tree/hurd/libdiskfs/lookup.c:149
#7  0x01037fd3 in diskfs_S_dir_lookup (dircred=0x8065440, path=0x12adf01 "n", 
flags=9, mode=0, retry=0x12abee4, retryname=0x12abeec "",
    returned_port=0x12ac2f0, returned_port_poly=0x12abde0)
    at 
/var/tmp/hurd-io+stat+fatfs/hurd-20060825/build-tree/hurd/libdiskfs/dir-lookup.c:128
#8  0x01040c9b in _Xdir_lookup (InHeadP=0x12aded0, OutHeadP=0x12abec0) at 
fsServer.c:1521
#9  0x0103e99d in diskfs_fs_server (InHeadP=0x12aded0, OutHeadP=0x805e7c8) at 
fsServer.c:2682
#10 0x0103718e in diskfs_demuxer (inp=0x12aded0, outp=0x12abec0)
    at 
/var/tmp/hurd-io+stat+fatfs/hurd-20060825/build-tree/hurd/libdiskfs/demuxer.c:32
#11 0x010aedfd in internal_demuxer (inp=0x12aded0, outheadp=0x12abec0)
    at 
/var/tmp/hurd-io+stat+fatfs/hurd-20060825/build-tree/hurd/libports/manage-multithread.c:119
#12 0x010dca9c in mach_msg_server_timeout () from /lib/libc.so.0.3
#13 0x010aeb84 in thread_function.7379 () at 
/var/tmp/hurd-io+stat+fatfs/hurd-20060825/build-tree/hurd/libports/manage-multithread.c:170
#14 0x010aeca8 in ports_manage_port_operations_multithread (bucket=0x805e688, 
demuxer=0x1037140 <diskfs_demuxer>, thread_timeout=120000,
    global_timeout=600000, hook=0) at 
/var/tmp/hurd-io+stat+fatfs/hurd-20060825/build-tree/hurd/libports/manage-multithread.c:198
#15 0x01045c56 in master_thread_function (demuxer=0x1037140)
    at 
/var/tmp/hurd-io+stat+fatfs/hurd-20060825/build-tree/hurd/libdiskfs/init-first.c:37
#16 0x010a892d in cthread_body (self=0x805e7c8) at 
/var/tmp/hurd-io+stat+fatfs/hurd-20060825/build-tree/hurd/libthreads/cthreads.c:306
#17 0x00000000 in ?? ()
#v-


Regards,
 Thomas

git.patch
Description: Text document

signature.asc
Description: Digital signature