Re: so-called pipe files (sh-np-*) do not get deleted when processes close.

[Michael Felt]

On 17/03/2021 23:12, Chet Ramey wrote:
> On 3/17/21 3:29 PM, Michael Felt wrote:
> > I tried as many combinations of commands as I could - and it seems 
> > that the regular behavior of dup2 on the opened fifo is enough to 
> > maintain communication.
>
> It's not, since FIFOs exist in the file system and have to be 
> available to
> open(2) when the other process (consumer, producer) wants to use them.
I didn't expect it to be perfect (see thx etc below). Better: I needed 
help in telling when it would most like fail! :)
> > Going into a system test (ie. a normal AdoptOpemJDK build process) 
> > that has nearly 3500 commands, each with two process_substitution 
> > commands.
>
> Consider the following scenario. You want to perform a regression test of
> two versions of a program, each of which produces textual output. You run
>
> diff <(program-version-1 args) <(program-version-2 args)

Yes, something to test. Thx. The ojdk scenario is: /usr/bin/printf > 
>(tee -a stdout.log) 2> >(tee -a stderr.log).

So, yes, in this case it is working because printf is the parent - 
(which I never seemed to find actually calling open() of the file. It 
seems to be using the fd opened by the child - in a magical way).

> This is defined to provide `diff' with two arguments. Let's call them
>
> /var/tmp/sh-np12345
> and
> /var/tmp/sh-np67890
>
> So diff runs, sees two arguments, opens both files, and does its thing.
> Diff has to see two filenames when it runs, otherwise it's an error.
But what I thoght I was seeing is that diff is the PARENT calling 
substitute_process() that create(s) a child process that reads/writes to 
a fifo file.
a) the child process never returns - it `exits` via, iirc, 
sh_exit(result) and the end of the routine
b) the parent gets the filename (pathname) - but I never see it actually 
opening it - only (when using bash -x) seeing the name in the -x command 
expansion.
> Now, let's say your change is there. The shell still runs
>
> diff /var/tmp/sh-np12345 /var/tmp/sh-np67890
>
> but, depending on how processes get scheduled, the shell forked to run
> the process substitutions has already unlinked those FIFOs. Diff will
> error out. The user will be unhappy. I will get bug reports.
>
> You have introduced a race condition. You may not get hit by it, but
> you cannot guarantee that no one will.
No I cannot - and for now it is a `hack` to solve a bigger issue. With 
3500 calls in a single build I hope the race occurs - and I'll finally 
see where the PARENT actually uses the name returned.
> The shell can't unlink the FIFO until it can guarantee that the
> processes that need to open it have opened it, and it can't guarantee
> that in the general case. It has to wait until the process completes,
> at least, and even that might not be correct.
Again, my issue was with >(command) substitution - where the `files` get 
written to by the parent - rather than reading them.

p.s. it is not my call to ask why they do not use regular redirection or 
pipes. Feels much simpler - but some people cannot miss the opportunity 
to use something blinky and shiney.

p.p.s. - If you have `words of wisdom` re: why this approach is much 
better than `standard` redirection - I am all ears!

*** Thanks again for the time to reply ***

> That's why the last-ditch approach is to remove all remaining FIFOs
> when the shell exits.
>
> > btw: other than the one open in the middle of process_substitution() 
> > I did not see anywhere where another process even tries to open the 
> > file.
>
> They are not necessarily shell processes, but what if they were? Since a
> FIFO is an object in the file system, you can just open(2) it. That's
> ostensibly the advantage of FIFOs.
>
> > what I also noticed is that the process, (iirc) that opens the file - 
> > never 'returns' - it ends via sh_exit() and the end of the routine.
>
> Of course. It's a process that is forked to run the command specified in
> the process substitution. What else does it need to do?

OpenPGP_0x722BFDB61F396FC2.asc
Description: OpenPGP public key

OpenPGP_signature
Description: OpenPGP digital signature