[Gluster-devel] 1.3.12 segfault

[Matt McCowan]

Greetings. It's my first post to this list so please bear with me while I try 
and flesh out the segfault I saw yesterday ...

Call me brave, call me stupid - without enough equipment on which to test 
things I have plunged glusterfs 1.3.12 straight into production on a small 
Opteron based cluster. The 14 clients are either 2 or 4 way Opteron driven (44 
core all up) running on amd64 Gentoo with a 2.6.20 kernel and the Gluster 2.7.3 
fuse module.
Running the same Gentoo as the clients the two servers are 4 way Opteron, dual 
homed (GigE) with a glusterfsd per network connection, sharing out 250G per 
daemon.

Yesterday the glusterfs process on one of the 2 way clients went to 100%. 
Attaching an strace to it showed it repeatedly calling nanosleep. Since the 
machine needed to be back online quickly (oh for the budget of LANL!) I tried 
to ctrl-c the strace, then sigterm, then had to sigkill it.

The sigterm must have got through to the glusterfs process because the log on 
the client contains:
"2009-01-14 14:01:53 W [glusterfs.c:416:glusterfs_cleanup_and_exit] glusterfs: 
shutting down server"
There were no log entries made when it was running at 100%.
The problem on the client was first noticed when a user tried to tab-complete a 
directory listing of the gluster mounted file system.

The gluster client was restarted. It was only a couple of hours later when some 
of the users reported issues that I noticed one of the glusterfsd's had died on 
a server. The glusterfsd segfault on the server coincides with killing the 
glusterfs on the client.

I haven't compiled gluster with debug, so following are entries from the server 
logs, client config, and a backtrace of the core dump (which unfortunately 
mirrors what's in the logs).

Side note: in an earlier 1.3.12 config we were running stripe across two 
glusterfsd backends. It proved to be quite unstable (specifically with 
directories sometimes not sync'ing on the backends) compared to the 
unify+namespace config. Otherwise glusterfs seems to be all round easier to 
install and use compared to my first cluster filesystem attempt with PVFS.


Contents of /var/log/glusterfsd.log:
====================================
2009-01-14 14:01:53 E [protocol.c:271:gf_block_unserialize_transport] server: 
EOF from peer (172.17.231.162:1016)
2009-01-14 14:01:53 E [protocol.c:271:gf_block_unserialize_transport] server: 
EOF from peer (172.17.231.162:1017)

TLA Repo Revision: glusterfs--mainline--2.5--patch-797
Time : 2009-01-14 14:01:53
Signal Number : 11

glusterfsd -f /etc/glusterfs/glusterfs-server-shareda.vol -l 
/var/log/glusterfs/glusterfsd.log -L WARNING
volume server
  type protocol/server
  option auth.ip.nsbricka.allow *
  option auth.ip.hans.allow *
  option auth.ip.data.allow *
  option bind-address 172.17.231.170
  option transport-type tcp/server
  subvolumes data hans nsbricka 
end-volume

volume data
  type performance/io-threads
  option cache-size 128M
  option thread-count 4
  subvolumes databrick 
end-volume

volume databrick
  type storage/posix
  option directory /var/local/shareda
end-volume

volume hans
  type cluster/afr
  subvolumes nsbricka nsbrickb 
end-volume

volume nsbrickb
  type protocol/client
  option remote-subvolume nsbricka
  option remote-host maelstroma9
  option transport-type tcp/client
end-volume

volume nsbricka
  type storage/posix
  option directory /var/local/namespace
end-volume

frame : type(0) op(0)
frame : type(0) op(0)

2009-01-14 14:01:53 E [protocol.c:271:gf_block_unserialize_transport] server: 
EOF from peer (172.17.231.162:1015)
/lib/libc.so.6[0x2af3d0e0f940]
/usr/lib64/glusterfs/1.3.12/xlator/cluster/afr.so(afr_close+0x140)[0x2aaaaacd37d0]
/usr/lib64/glusterfs/1.3.12/xlator/protocol/server.so(server_protocol_cleanup+0x1af)[0x2aaaaaef80cf]
/usr/lib64/glusterfs/1.3.12/xlator/protocol/server.so(notify+0x6e)[0x2aaaaaef853e]
/usr/lib/libglusterfs.so.0(transport_unref+0x64)[0x2af3d0ab32b4]
/usr/lib64/glusterfs/1.3.12/transport/tcp/client.so(tcp_disconnect+0x7d)[0x2aaaaaffdcfd]
/usr/lib64/glusterfs/1.3.12/xlator/protocol/server.so(notify+0x61)[0x2aaaaaef8531]
/usr/lib/libglusterfs.so.0(sys_epoll_iteration+0xbb)[0x2af3d0ab3c4b]
/usr/lib/libglusterfs.so.0(poll_iteration+0x78)[0x2af3d0ab3008]
[glusterfs](main+0x67c)[0x40288c]
/lib/libc.so.6(__libc_start_main+0xf4)[0x2af3d0dfd374]
[glusterfs][0x401d59]
---------
====================================
end of glusterfsd.log


/etc/glusterfs/glusterfs-client.vol:
====================================
volume brick1
        type protocol/client
        option transport-type tcp/client     # for TCP/IP transport
        option remote-host maelstroma0
        option transport-timeout 120
        option remote-subvolume data # name of the remote volume
end-volume

volume brick2
        type protocol/client
        option transport-type tcp/client     # for TCP/IP transport
        option remote-host maelstroma0a
        option transport-timeout 120
        option remote-subvolume data # name of the remote volume
end-volume

volume brick3
        type protocol/client
        option transport-type tcp/client     # for TCP/IP transport
        option remote-host maelstroma9
        option transport-timeout 120
        option remote-subvolume data # name of the remote volume
end-volume

volume brick4
        type protocol/client
        option transport-type tcp/client     # for TCP/IP transport
        option remote-host maelstroma9a
        option transport-timeout 120
        option remote-subvolume data # name of the remote volume
end-volume

volume ns
        type protocol/client
        option transport-type tcp/client
        option remote-host gluster
        option transport-timeout 120
        option remote-subvolume hans
end-volume

volume unify
        type cluster/unify
        option scheduler rr
        option rr.limits.min-free-disk 5
        option namespace ns
        subvolumes brick1 brick2 brick3 brick4
end-volume

volume iothreads
        type performance/io-threads
        #option thread-count 8
        option thread-count 4
        option cache-size 64M
        subvolumes unify
end-volume

volume readahead
        type performance/read-ahead
        option page-size 1024kb
        option page-count 10
        subvolumes iothreads
end-volume

volume iocache
        type performance/io-cache
        option cache-size 64MB #default 32M
        option page-size 1MB #default 128kb
        subvolumes readahead
end-volume

volume writebehind
        type performance/write-behind
        option aggregate-size 1MB
        option flush-behind off
        subvolumes iocache
end-volume
====================================
end of glusterfs-client.vol

gdb backtrace:
====================================
gdb /usr/sbin/glusterfsd /core.28935 
GNU gdb 6.7.1
Copyright (C) 2007 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu"...
(no debugging symbols found)
Using host libthread_db library "/lib/libthread_db.so.1".
Reading symbols from /usr/lib64/libglusterfs.so.0...(no debugging symbols 
found)...done.
Loaded symbols for /usr/lib/libglusterfs.so.0
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libdl.so.2
Reading symbols from /lib64/libpthread.so.0...done.
Loaded symbols for /lib/libpthread.so.0
Reading symbols from /lib64/libc.so.6...done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /usr/lib64/glusterfs/1.3.12/xlator/storage/posix.so...done.
Loaded symbols for /usr/lib64/glusterfs/1.3.12/xlator/storage/posix.so
Reading symbols from 
/usr/lib64/glusterfs/1.3.12/xlator/protocol/client.so...done.
Loaded symbols for /usr/lib64/glusterfs/1.3.12/xlator/protocol/client.so
Reading symbols from /usr/lib64/glusterfs/1.3.12/xlator/cluster/afr.so...done.
Loaded symbols for /usr/lib64/glusterfs/1.3.12/xlator/cluster/afr.so
Reading symbols from 
/usr/lib64/glusterfs/1.3.12/xlator/performance/io-threads.so...done.
Loaded symbols for /usr/lib64/glusterfs/1.3.12/xlator/performance/io-threads.so
Reading symbols from 
/usr/lib64/glusterfs/1.3.12/xlator/protocol/server.so...done.
Loaded symbols for /usr/lib64/glusterfs/1.3.12/xlator/protocol/server.so
Reading symbols from /usr/lib64/glusterfs/1.3.12/transport/tcp/client.so...done.
Loaded symbols for /usr/lib64/glusterfs/1.3.12/transport/tcp/client.so
Reading symbols from /usr/lib64/glusterfs/1.3.12/transport/tcp/server.so...done.
Loaded symbols for /usr/lib64/glusterfs/1.3.12/transport/tcp/server.so
Reading symbols from /usr/lib64/glusterfs/1.3.12/auth/ip.so...done.
Loaded symbols for /usr/lib64/glusterfs/1.3.12/auth/ip.so
Reading symbols from /lib64/libnss_files.so.2...done.
Loaded symbols for /lib/libnss_files.so.2
Reading symbols from /lib64/libnss_dns.so.2...done.
Loaded symbols for /lib/libnss_dns.so.2
Reading symbols from /lib64/libresolv.so.2...done.
Loaded symbols for /lib/libresolv.so.2
Reading symbols from /lib64/libgcc_s.so.1...done.
Loaded symbols for /lib/libgcc_s.so.1

Core was generated by `[glusterfs]                                              
'.
Program terminated with signal 11, Segmentation fault.
#0  0x00002aaaaacd37d0 in afr_close ()
   from /usr/lib64/glusterfs/1.3.12/xlator/cluster/afr.so
(gdb) q
====================================
end of bactrace


Thanks for glusterfs
Regards
Matt McCowan
sysadmin
RPS MetOcean
Perth, Western Australia