Re: so-called pipe files (sh-np-*) do not get deleted when processes close.

[Chet Ramey]

On 3/18/21 5:53 AM, Michael Felt wrote:

> Yes, something to test. Thx. The ojdk scenario is: /usr/bin/printf > >(tee 
> -a stdout.log) 2> >(tee -a stderr.log).
>
> So, yes, in this case it is working because printf is the parent - (which I 
> never seemed to find actually calling open() of the file. It seems to be 
> using the fd opened by the child - in a magical way).

It's the redirection. The shell does the open, since the filename resulting
from process substitution is the target of a redirection operator. This is
a common idiom -- so common, in fact, that people have interpreted it to
mean that the entire `> >(xxx)' is a single operator.

However, the shell expands redirections in the child process it forks to
exec printf, so that child shell is what does the process substitution.
That might be the problem here.

The command itself doesn't do anything, though. `tee' just sits there
waiting for data to write to log files. It has no purpose. I'm not sure
what the intent is.

If you wrapped that command into a script, it's unlikely that either `tee'
would exit (why would they?) before `printf' exits and the script
completes. In bash-5.0, there would be nothing to remove the FIFOs.


> > This is defined to provide `diff' with two arguments. Let's call them
> >
> > /var/tmp/sh-np12345
> > and
> > /var/tmp/sh-np67890
> >
> > So diff runs, sees two arguments, opens both files, and does its thing.
> > Diff has to see two filenames when it runs, otherwise it's an error.
> But what I thoght I was seeing is that diff is the PARENT calling 
> substitute_process() that create(s) a child process that reads/writes to a 
> fifo file.

Yes and no. Process substitution is a word expansion that results in
a filename. The stuff between the parens defines a command that writes to
or reads from a pipe expressed as a filename (/dev/fd/NN or a FIFO) that is
the result of the word expansion. In this case, the process substitution is
the target of a redirection, so the shell performs that word expansion
before it execs diff.

> a) the child process never returns - it `exits` via, iirc, sh_exit(result) 
> and the end of the routine

It executes the specified command and exits.


> b) the parent gets the filename (pathname) - but I never see it actually 
> opening it - only (when using bash -x) seeing the name in the -x command 
> expansion.

It doesn't have to. The filename itself is the expansion: it's an object
you can use to communicate with an asynchronous process. This is how you
can have programs that expect a filename use program-generated output, for
instance, without using a temp file.

In this case, it opens the FIFO because it is the target of a redirection.

> > Now, let's say your change is there. The shell still runs
> >
> > diff /var/tmp/sh-np12345 /var/tmp/sh-np67890
> >
> > but, depending on how processes get scheduled, the shell forked to run
> > the process substitutions has already unlinked those FIFOs. Diff will
> > error out. The user will be unhappy. I will get bug reports.
> >
> > You have introduced a race condition. You may not get hit by it, but
> > you cannot guarantee that no one will.
> No I cannot - and for now it is a `hack` to solve a bigger issue. With 3500 
> calls in a single build I hope the race occurs - and I'll finally see where 
> the PARENT actually uses the name returned.

You mean in terms of using the filename as an argument to a shell builtin?
Otherwise you'll have to trace into other child process execution.

> > The shell can't unlink the FIFO until it can guarantee that the
> > processes that need to open it have opened it, and it can't guarantee
> > that in the general case. It has to wait until the process completes,
> > at least, and even that might not be correct.
> Again, my issue was with >(command) substitution - where the `files` get 
> written to by the parent - rather than reading them.

It doesn't matter. Let's try that scenario. A FIFO reader can live forever;
just waiting for someone to open the FIFO to write to it. In this case,
the child process opens the FIFO for read, and blocks until another
process opens it for write. That's the shell, since it's the target of a
redirection, but it doesn't have to be (the filename could just be passed
to another process as an argument). The file descriptor gets passed to
printf as its stdout (and printf apparently does nothing with it) and then
closed as part of the process exiting. When that happens, the tee should
get EOF and exit. The shell notices that tee exits and cleans up the FIFO.
If the shell exits, for instance, before the tee exits, nothing cleans up 
the FIFO.

> p.s. it is not my call to ask why they do not use regular redirection or 
> pipes. Feels much simpler - but some people cannot miss the opportunity to 
> use something blinky and shiney.
>
> p.p.s. - If you have `words of wisdom` re: why this approach is much better 
> than `standard` redirection - I am all ears!

If you want to send the output to the terminal (or wherever) as well as a
log file, something like `tee' is required. If you want to keep stderr and
stdout logs separate, two separate redirections are required. If you want
this to happen asynchronously, process substitution is required.

Otherwise, you introduce temporary files and synchronous behavior when you
want neither.

If it helps, the above printf command is syntactic sugar for something like
(with no error checking):

{
/usr/bin/printf >tmpout 2>tmperr
cat tmperr tmpout

cat tmpout >> stdout.log
cat tmperr >> stderr.log

rm -f tmpout tmperr
}


--
``The lyf so short, the craft so long to lerne.'' - Chaucer
                 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRU    chet@case.edu    http://tiswww.cwru.edu/~chet/