coreutils
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [PATCH] tee: Add --pipe-check to allow instantly detecting closed ou


From: Pádraig Brady
Subject: Re: [PATCH] tee: Add --pipe-check to allow instantly detecting closed outputs
Date: Mon, 28 Nov 2022 22:43:47 +0000
User-agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.0

On 19/11/2022 19:27, Arsen Arsenović wrote:

Pádraig Brady <P@draigBrady.com> writes:

Thanks a lot for the patch.

There is similar functionality in tail.c (which should probably be reused
if we do decide to implement this in tee as it's tricky to do portably).
Anyway the tail(1) case makes sense considering:

Looking at ``check_output_alive'', this logic should be fairly easy to
reuse.  Worst case, I'd need to refactor some of the code to handle the
non-pipe-check case more elegantly so that it doesn't become too
unreadable when also adding the logic to pick between poll and select.

   tail -f file.log | grep -q trigger && process_immediately

So a similar use case with tee might be:

   tee -p <file.in >(grep -q trigger_1) | grep -q trigger_2 && 
process_immediately

However tee wouldn't be waiting for more input in that case.
It would either consume the whole file, or exit when processing it.

So a more appropriate case is:

   intermittent_output |
   tee -p >(grep -q trigger_1) | grep -q trigger_2 && process_immediately

This case is just about what we were using tee for when I wrote this
patch.  We use a compiler wrapper that compares the outputs of few
compilers by running them in parallel (to detect some new behavior), and
we started noticing some builds would mysteriously hang forever.  I
traced it back to ``tee'' waiting on stdin even after the child
processes die on occasion.

Really, that was an oversight on my part.  I didn't think of the (quite
normal) case of stdin being some long-lived low traffic pipe rather than
either a file or /dev/null or a pipe that gets closed properly after
writing.

Where intermittent_output is stdin, the use case is a bit contrived
as any input will cause tee to exit.
The general more general non stdin case above has some merit,
though not as common as the tail example I think.

Right, but no user input could ever happen when the hangs described
above happened, since it'd happen in some automated code, so (AFAICT)
the only option was to teach tee to detect outputs going away more
promptly.

To get a better handle on the generality of this
I went back about 16 years on the list,
and even though there have been many tee functionality requests over the years,
this is the first time this has been requested.

Trying to understand your use case better,
I'm presuming the input generator is depending on the compiler runs
(written to by tee) to exit cleanly, before exiting / generating more?
Hence the hangs?

If that was the case then there still might be the potential for hangs
even if tee detected closed pipes. I.e. if the compiler runs hung rather
than exited, this not be distinguishable from tee as the pipe outputs would 
remain.

If that's the case this has become a bit less generally useful in my mind.

To keep tee data driven, perhaps your input could periodically
send a "clock" input through to (say a newline), to check everything
is still running as expected.  I.e. your periodic input generator
seems like it would be async already, so it would be better to
add any extra async logic there, and keep tee more simple and data driven.

cheers,
Pádraig.

p.s. while reading the tee docs in this area, they were a bit dispersed,
so the attached updates the texinfo for the -p option to summarise
it's operation and how it differs from the default operation.

Attachment: tee-p-describe.patch
Description: Text Data


reply via email to

[Prev in Thread] Current Thread [Next in Thread]