Re: bash leaks sh-np-NNN files and pipes in /tmp/ when command substitution is used

[Yuri]

On 12/10/2013 23:29, Yuri wrote:
> Some of my scripts use command substitution, and now I see that there 
> are lots of files like these in /tmp:
> prw-------   1 yuri   wheel          0 Dec 10 13:32 sh-np-1386738492
> -rw-r--r--   1 yuri   wheel    3278909 Dec 10 14:54 sh-np-1386721176
>
> Besides the obvious question why they aren't deleted in the end, two 
> questions:
> Shouldn't bash delete the pipe files once the child process opened 
> them? On most platforms the deleted pipe will function the same with 
> the benefit that it will get deleted automatically.
> Why some of he files /tmp/sh-np-NNN are regular files, not pipes? When 
> I look through code I see mkfifo call creates them. Shouldn't it 
> always create fifos and not files? 

Ok, I figured the details out.
The actual script that I run is this radio playing command:
stdbuf -i 0 -o 0 -e 0 mplayer 
http://kqed-ice.streamguys.org:80/kqedradio-ch-e1 | \
  stdbuf -i 0 -o 0 -e 0 tee >(grep "^XXX XXX" | prepend-time >> test.log)

prepend-time is a script:
gawk '{ print strftime("%Y-%m-%d %H:%M:%S"), $0; }'

The problem only occurs with stdbuf(1) commands inserted there, and not 
without them.
1: bash creates a FIFO /tmp/sh-np-NNNNN
2: bash unlinks this FIFO
3: tee is passed /tmp/sh-np-NNNNN as an argument, and it creates a 
regular file with the same name

This is the race condition.
Obviously, bash doesn't ensure that unlink is issued after the child 
command opons it, and not before that. Order of operations 2 and 3 
should be reversed.
And just waiting for a while isn't enough! bash should only unlink the 
fifo after it has been open by the child command.

Please also note, that this is on FreeBSD, which behavior might be 
different. Linux or other OSes might be getting lucky here with timing, 
not sure. FreeBSD also has its own version of stdbuf(1).
This bug needs to be fixed.

The original intention of stdbuf there was to make it all unbufferer. 
Whether this works or not isn't clear, but it doesn't matter for the 
purpose of this bug report.

Suggested solution:
On FreeBSD, for example, there is the kqueue(2) facility, which is able 
to wait on the event of "link-change": EVFILT_VNODE operation (takes 
file descriptor) allows sub-operation NOTE_LINK (The link count on the 
file changed)
So the sequence will be like this:
1. bash creates FIFO
2. bash opens FIFO
3. bash waits with kqueue(2) with EVFILT_VNODE/NOTE_LINK
4. bash launches the child command
5. child command opens the FIFO as stdio
6. bash catches the link-change event
7. bash unlinks FIFO

Linux and other OSes must have an equivalent facility.
Sounds like a fun project to do. -)
Absolute must-have if bash needs to support command substitution.

Yuri

CCing to the maintainer of bash on FreeBSD