Hang in bgp_delete

[Graham Northup]

Configuration Information:
Machine: x86_64
OS: linux-gnu
Compiler: gcc
Compilation CFLAGS:  -DPROGRAM='bash' -DCONF_HOSTTYPE='x86_64'
-DCONF_OSTYPE='linux-gnu' -DCONF_MACHTYPE='x86_64-unknown-linux-gnu'
-DCONF_VENDOR='unknown' -DLOCALEDIR='/usr/share/locale' -DPACKAGE='bash'
-DSHELL -DHAVE_CONFIG_H   -I.  -I. -I./include -I./lib
-D_FORTIFY_SOURCE=2 -march=x86-64 -mtune=generic -O2 -pip
e -fstack-protector-strong -g -fvar-tracking-assignments -g
-fvar-tracking-assignments
-DDEFAULT_PATH_VALUE='/usr/local/sbin:/usr/local/bin:/usr/bin'
-DSTANDARD_UTILS_PATH='/usr/bin' -DSYS_BASHRC='/etc/bash.bashrc'
-DSYS_BASH_LOGOUT='/etc/bash.bash_logout' -Wno-parentheses
-Wno-format-security
uname output: Linux gmx 4.8.13-1-ARCH #1 SMP PREEMPT Fri Dec 9 07:24:34
CET 2016 x86_64 GNU/Linux
Machine Type: x86_64-unknown-linux-gnu

Bash Version: 4.4
Patch Level: 11
Release Status: release

Description:

I'm getting a mysterious hang on one of our Arch Linux machines for a
particular, rather simple script; getting a debugger attached to the
process after building some debugging symbols, I tracked the hang down
to this loop in bgp_delete (with some minor formatting):

for (
   psi = *(pshash_getbucket (pid));
   psi != NO_PIDSTAT;
   psi = bgpids.storage[psi].bucket_next
)
    if (bgpids.storage[psi].pid == pid)
        break;

...the problem is, according to my debugger:

(gdb) p psi
$1 = 11506
(gdb) p bgpids.storage[psi].bucket_next
$2 = 11506

...and so this just sits there wedging a core :)

I'm not entirely sure what circumstances cause this, but it feels pretty
racy; it takes, on average, a couple days to get this machine to
reliably repeat the issue. I'll leave this process alive for now if
you'd like me to gather more forensics. (I do have a core dump, but it's
~5.5MB :)

For posterity, and reference below, here's a backtrace--sorry that my UA
tries to word wrap it:

#0  0x000000000043ff0e in bgp_delete (pid=pid@entry=15980) at jobs.c:868
#1  0x0000000000443f89 in make_child (command=0xb6b930
"/manage/totaldisk.sh", async_p=async_p@entry=1) at jobs.c:2093
#2  0x000000000042ff9c in execute_simple_command
(simple_command=0x93bd10, pipe_in=pipe_in@entry=-1,
pipe_out=pipe_out@entry=-1, async=async@entry=1,
fds_to_close=fds_to_close@entry=0xbbc4c0) at execute_cmd.c:4088
#3  0x0000000000431e5c in execute_command_internal (command=0x93bce0,
asynchronous=asynchronous@entry=1, pipe_in=pipe_in@entry=-1,
pipe_out=pipe_out@entry=-1, fds_to_close=fds_to_close@entry=0xbbc4c0) at
execute_cmd.c:802
#4  0x0000000000433176 in execute_connection (fds_to_close=0xbbc4c0,
pipe_out=-1, pipe_in=-1, asynchronous=1, command=0x93bd60) at
execute_cmd.c:2576
#5  execute_command_internal (command=command@entry=0x93bd60,
asynchronous=asynchronous@entry=1, pipe_in=pipe_in@entry=-1,
pipe_out=pipe_out@entry=-1, fds_to_close=fds_to_close@entry=0xbbc4c0) at
execute_cmd.c:971
#6  0x0000000000433105 in execute_connection (fds_to_close=0xbbc4c0,
pipe_out=-1, pipe_in=-1, asynchronous=1, command=0x93be70) at
execute_cmd.c:2564
#7  execute_command_internal (command=command@entry=0x93be70,
asynchronous=asynchronous@entry=1, pipe_in=pipe_in@entry=-1,
pipe_out=pipe_out@entry=-1, fds_to_close=fds_to_close@entry=0xbbc4c0) at
execute_cmd.c:971
#8  0x0000000000433105 in execute_connection (fds_to_close=0xbbc4c0,
pipe_out=-1, pipe_in=-1, asynchronous=1, command=0x93bfe0) at
execute_cmd.c:2564
#9  execute_command_internal (command=command@entry=0x93bfe0,
asynchronous=asynchronous@entry=1, pipe_in=pipe_in@entry=-1,
pipe_out=pipe_out@entry=-1, fds_to_close=fds_to_close@entry=0xbbc4c0) at
execute_cmd.c:971
#10 0x0000000000433105 in execute_connection (fds_to_close=0xbbc4c0,
pipe_out=-1, pipe_in=-1, asynchronous=1, command=0x93c0f0) at
execute_cmd.c:2564
#11 execute_command_internal (command=command@entry=0x93c0f0,
asynchronous=asynchronous@entry=1, pipe_in=pipe_in@entry=-1,
pipe_out=pipe_out@entry=-1, fds_to_close=fds_to_close@entry=0xbbc4c0) at
execute_cmd.c:971
#12 0x0000000000433105 in execute_connection (fds_to_close=0xbbc4c0,
pipe_out=-1, pipe_in=-1, asynchronous=1, command=0x93c200) at
execute_cmd.c:2564
#13 execute_command_internal (command=command@entry=0x93c200,
asynchronous=asynchronous@entry=1, pipe_in=pipe_in@entry=-1,
pipe_out=pipe_out@entry=-1, fds_to_close=fds_to_close@entry=0xbbc4c0) at
execute_cmd.c:971
#14 0x0000000000433105 in execute_connection (fds_to_close=0xbbc4c0,
pipe_out=-1, pipe_in=-1, asynchronous=0, command=0x93c370) at
execute_cmd.c:2564
#15 execute_command_internal (command=command@entry=0x93c370,
asynchronous=asynchronous@entry=0, pipe_in=pipe_in@entry=-1,
pipe_out=pipe_out@entry=-1, fds_to_close=fds_to_close@entry=0xbbc4c0) at
execute_cmd.c:971
#16 0x0000000000433a7e in execute_command (command=0x93c370) at
execute_cmd.c:405
#17 0x0000000000433b2f in execute_while_or_until
(while_command=0x93c3a0, type=type@entry=0) at execute_cmd.c:3509
#18 0x0000000000431cad in execute_while_command
(while_command=<optimized out>) at execute_cmd.c:3450
#19 execute_command_internal (command=command@entry=0x93c3c0,
asynchronous=asynchronous@entry=0, pipe_in=pipe_in@entry=-1,
pipe_out=pipe_out@entry=-1, fds_to_close=fds_to_close@entry=0x93c3f0) at
execute_cmd.c:911
#20 0x0000000000433a7e in execute_command (command=0x93c3c0) at
execute_cmd.c:405
#21 0x000000000041c4a2 in reader_loop () at eval.c:180
#22 0x000000000041b1c2 in main (argc=2, argv=0x7fff79e96c78,
env=0x7fff79e96c90) at shell.c:792

...and here is xxd /proc/3127/cmdline--the hanging process. I was asked
by gdb when I dumped core to note that there are embedded NULs in here:

00000000: 2f62 696e 2f62 6173 6800 2f6d 616e 6167  /bin/bash./manag
00000010: 652f 7275 6e2e 7368 00                   e/run.sh.

Repeat-By:

Attached is a tarball with the current master of (1) the script 3127 was
running, (2) all of the scripts and program sources it calls, and (3) a
systemd service which invoked 3127. Assuming your build is susceptible
to the bug--exact conditions are quite unclear--extract this tarball's
contents, rename the directory in its root to "/manage", and execute
"/bin/bash /manage/run.sh"--or, more faithfully, install manage.service
into a systemd unit path of your choosing and start that service.

Let me know if there's anything else I can provide :)

Thanks,
Graham

manage3client-master-5e9652e7eb2f53545244aa180427f019ca3e92d6.tar.bz2
Description: Binary data

signature.asc
Description: OpenPGP digital signature