Re: so-called pipe files (sh-np-*) do not get deleted when processes close.

[Chet Ramey]

On 3/17/21 3:29 PM, Michael Felt wrote:
> I tried as many combinations of commands as I could - and it seems that the 
> regular behavior of dup2 on the opened fifo is enough to maintain 
> communication.

It's not, since FIFOs exist in the file system and have to be available to
open(2) when the other process (consumer, producer) wants to use them.

> Going into a system test (ie. a normal AdoptOpemJDK build process) that has 
> nearly 3500 commands, each with two process_substitution commands.

Consider the following scenario. You want to perform a regression test of
two versions of a program, each of which produces textual output.  You run

diff <(program-version-1 args) <(program-version-2 args)

This is defined to provide `diff' with two arguments. Let's call them

/var/tmp/sh-np12345
and
/var/tmp/sh-np67890

So diff runs, sees two arguments, opens both files, and does its thing.
Diff has to see two filenames when it runs, otherwise it's an error.

Now, let's say your change is there. The shell still runs

diff /var/tmp/sh-np12345 /var/tmp/sh-np67890

but, depending on how processes get scheduled, the shell forked to run
the process substitutions has already unlinked those FIFOs. Diff will
error out. The user will be unhappy. I will get bug reports.

You have introduced a race condition. You may not get hit by it, but
you cannot guarantee that no one will.

The shell can't unlink the FIFO until it can guarantee that the
processes that need to open it have opened it, and it can't guarantee
that in the general case. It has to wait until the process completes,
at least, and even that might not be correct.

That's why the last-ditch approach is to remove all remaining FIFOs
when the shell exits.

> btw: other than the one open in the middle of process_substitution() I did 
> not see anywhere where another process even tries to open the file.

They are not necessarily shell processes, but what if they were? Since a
FIFO is an object in the file system, you can just open(2) it. That's
ostensibly the advantage of FIFOs.

> what I also noticed is that the process, (iirc) that opens the file - never 
> 'returns' - it ends via sh_exit() and the end of the routine.

Of course. It's a process that is forked to run the command specified in
the process substitution. What else does it need to do?

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
                 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRU    chet@case.edu    http://tiswww.cwru.edu/~chet/