[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: so-called pipe files (sh-np-*) do not get deleted when processes clo

From: Michael Felt
Subject: Re: so-called pipe files (sh-np-*) do not get deleted when processes close.
Date: Sat, 20 Mar 2021 20:15:21 +0100
User-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.8.1

Scraping through this - thanks for the lessons aka explanations.

On 18/03/2021 16:08, Chet Ramey wrote:
On 3/18/21 5:53 AM, Michael Felt wrote:

Yes, something to test. Thx. The ojdk scenario is: /usr/bin/printf > >(tee -a stdout.log) 2> >(tee -a stderr.log).

So, yes, in this case it is working because printf is the parent - (which I never seemed to find actually calling open() of the file. It seems to be using the fd opened by the child - in a magical way).

It's the redirection. The shell does the open, since the filename resulting from process substitution is the target of a redirection operator. This is
a common idiom -- so common, in fact, that people have interpreted it to
mean that the entire `> >(xxx)' is a single operator.

However, the shell expands redirections in the child process it forks to
exec printf, so that child shell is what does the process substitution.
That might be the problem here.

The command itself doesn't do anything, though. `tee' just sits there
waiting for data to write to log files. It has no purpose. I'm not sure
what the intent is.

If you wrapped that command into a script, it's unlikely that either `tee'
would exit (why would they?) before `printf' exits and the script
completes. In bash-5.0, there would be nothing to remove the FIFOs.
If I understand correctly, the commands are generated by gmake as it processes targets.

This is defined to provide `diff' with two arguments. Let's call them


So diff runs, sees two arguments, opens both files, and does its thing.
Diff has to see two filenames when it runs, otherwise it's an error.

But what I thoght I was seeing is that diff is the PARENT calling substitute_process() that create(s) a child process that reads/writes to a fifo file.

Yes and no. Process substitution is a word expansion that results in
a filename. The stuff between the parens defines a command that writes to
or reads from a pipe expressed as a filename (/dev/fd/NN or a FIFO) that is the result of the word expansion. In this case, the process substitution is
the target of a redirection, so the shell performs that word expansion
before it execs diff.

a) the child process never returns - it `exits` via, iirc, sh_exit(result) and the end of the routine

It executes the specified command and exits.
Got it: must remember - initially it is bash busy with word expansion (a new bash child for each 'process substitution'

b) the parent gets the filename (pathname) - but I never see it actually opening it - only (when using bash -x) seeing the name in the -x command expansion.

It doesn't have to. The filename itself is the expansion: it's an object
you can use to communicate with an asynchronous process. This is how you
can have programs that expect a filename use program-generated output, for
instance, without using a temp file.
Yes - for me at least it is much easier to fathom as input - that ends and behaves/looks/feels like EOF.

In this case, it opens the FIFO because it is the target of a redirection.
Nods: just as it would if it was the output from a program than had just run 'moments' before.

Now, let's say your change is there. The shell still runs

diff /var/tmp/sh-np12345 /var/tmp/sh-np67890

but, depending on how processes get scheduled, the shell forked to run
the process substitutions has already unlinked those FIFOs. Diff will
error out. The user will be unhappy. I will get bug reports.

You have introduced a race condition. You may not get hit by it, but
you cannot guarantee that no one will.
No I cannot - and for now it is a `hack` to solve a bigger issue. With 3500 calls in a single build I hope the race occurs - and I'll finally see where the PARENT actually uses the name returned.

You mean in terms of using the filename as an argument to a shell builtin?
Otherwise you'll have to trace into other child process execution.

/usr/bin/printf is not a built-n (afaik)

If I understand correctly - from printf perspective we have

/usr/bin/printf "Some formatted message" > /tmp/sh-np.123456 2> /tmp/sh-np.9876543 &

And, if for ease of discussion we say program1 is PID-123456 and program is PID-987654 - these programs have no way of knowing their stdin is named /tmp/sh-np-something? BING: as the dutch (used to) say - the quarter drops - the other programs (e.g., tee) have no fifo knowledge - they are who/what they are. What maybe needed for this situation - is rather than directly execev() the program - yet another fork (for the execve() - and wait for that program to hit it's EOF on input and then the sleeping 'word expansion child' cleans up the fifo file it created for the communication path.

* Am I getting closer? :)

The shell can't unlink the FIFO until it can guarantee that the
processes that need to open it have opened it, and it can't guarantee
that in the general case. It has to wait until the process completes,
at least, and even that might not be correct.

Again, my issue was with >(command) substitution - where the `files` get written to by the parent - rather than reading them.

It doesn't matter. Let's try that scenario. A FIFO reader can live forever;
just waiting for someone to open the FIFO to write to it. In this case,
the child process opens the FIFO for read, and blocks until another
process opens it for write. That's the shell, since it's the target of a
redirection, but it doesn't have to be (the filename could just be passed
to another process as an argument). The file descriptor gets passed to
printf as its stdout (and printf apparently does nothing with it) and then
closed as part of the process exiting. When that happens, the tee should
get EOF and exit. The shell notices that tee exits and cleans up the FIFO. If the shell exits, for instance, before the tee exits, nothing cleans up the FIFO.

p.s. it is not my call to ask why they do not use regular redirection or pipes. Feels much simpler - but some people cannot miss the opportunity to use something blinky and shiney.

p.p.s. - If you have `words of wisdom` re: why this approach is much better than `standard` redirection - I am all ears!

If you want to send the output to the terminal (or wherever) as well as a
log file, something like `tee' is required. If you want to keep stderr and
stdout logs separate, two separate redirections are required. If you want
this to happen asynchronously, process substitution is required.

Otherwise, you introduce temporary files and synchronous behavior when you
want neither.
Below - not helping yet. And, in the case of the process I am trying to resolve the first command is:

/usr/bin/printf "Pretty message" >tmpout 2>tmpout "

That too, makes my head hurt. why not

/usr/bin/printf "Pretty message\n" 2>&1 | tee -a /my/special/logfile

(This is what I am also asking in an issue at AdoptOpenJDK)

If it helps, the above printf command is syntactic sugar for something like
(with no error checking):

/usr/bin/printf >tmpout 2>tmperr
cat tmperr tmpout

cat tmpout >> stdout.log
cat tmperr >> stderr.log

rm -f tmpout tmperr

Attachment: OpenPGP_0x722BFDB61F396FC2.asc
Description: OpenPGP public key

Attachment: OpenPGP_signature
Description: OpenPGP digital signature

reply via email to

[Prev in Thread] Current Thread [Next in Thread]