[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: so-called pipe files (sh-np-*) do not get deleted when processes clo

From: Michael Felt
Subject: Re: so-called pipe files (sh-np-*) do not get deleted when processes close.
Date: Thu, 18 Mar 2021 10:53:41 +0100
User-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.8.1

On 17/03/2021 23:12, Chet Ramey wrote:
On 3/17/21 3:29 PM, Michael Felt wrote:
I tried as many combinations of commands as I could - and it seems that the regular behavior of dup2 on the opened fifo is enough to maintain communication.

It's not, since FIFOs exist in the file system and have to be available to
open(2) when the other process (consumer, producer) wants to use them.
I didn't expect it to be perfect (see thx etc below). Better: I needed help in telling when it would most like fail! :)

Going into a system test (ie. a normal AdoptOpemJDK build process) that has nearly 3500 commands, each with two process_substitution commands.

Consider the following scenario. You want to perform a regression test of
two versions of a program, each of which produces textual output. You run

diff <(program-version-1 args) <(program-version-2 args)

Yes, something to test. Thx. The ojdk scenario is: /usr/bin/printf > >(tee -a stdout.log) 2> >(tee -a stderr.log).

So, yes, in this case it is working because printf is the parent - (which I never seemed to find actually calling open() of the file. It seems to be using the fd opened by the child - in a magical way).

This is defined to provide `diff' with two arguments. Let's call them


So diff runs, sees two arguments, opens both files, and does its thing.
Diff has to see two filenames when it runs, otherwise it's an error.

But what I thoght I was seeing is that diff is the PARENT calling substitute_process() that create(s) a child process that reads/writes to a fifo file. a) the child process never returns - it `exits` via, iirc, sh_exit(result) and the end of the routine b) the parent gets the filename (pathname) - but I never see it actually opening it - only (when using bash -x) seeing the name in the -x command expansion.
Now, let's say your change is there. The shell still runs

diff /var/tmp/sh-np12345 /var/tmp/sh-np67890

but, depending on how processes get scheduled, the shell forked to run
the process substitutions has already unlinked those FIFOs. Diff will
error out. The user will be unhappy. I will get bug reports.

You have introduced a race condition. You may not get hit by it, but
you cannot guarantee that no one will.
No I cannot - and for now it is a `hack` to solve a bigger issue. With 3500 calls in a single build I hope the race occurs - and I'll finally see where the PARENT actually uses the name returned.

The shell can't unlink the FIFO until it can guarantee that the
processes that need to open it have opened it, and it can't guarantee
that in the general case. It has to wait until the process completes,
at least, and even that might not be correct.

Again, my issue was with >(command) substitution - where the `files` get written to by the parent - rather than reading them.

p.s. it is not my call to ask why they do not use regular redirection or pipes. Feels much simpler - but some people cannot miss the opportunity to use something blinky and shiney.

p.p.s. - If you have `words of wisdom` re: why this approach is much better than `standard` redirection - I am all ears!

*** Thanks again for the time to reply ***

That's why the last-ditch approach is to remove all remaining FIFOs
when the shell exits.

btw: other than the one open in the middle of process_substitution() I did not see anywhere where another process even tries to open the file.

They are not necessarily shell processes, but what if they were? Since a
FIFO is an object in the file system, you can just open(2) it. That's
ostensibly the advantage of FIFOs.

what I also noticed is that the process, (iirc) that opens the file - never 'returns' - it ends via sh_exit() and the end of the routine.

Of course. It's a process that is forked to run the command specified in
the process substitution. What else does it need to do?

Attachment: OpenPGP_0x722BFDB61F396FC2.asc
Description: OpenPGP public key

Attachment: OpenPGP_signature
Description: OpenPGP digital signature

reply via email to

[Prev in Thread] Current Thread [Next in Thread]