Re: [PATCH] tee: Add --pipe-check to allow instantly detecting closed outputs

[Arsen Arsenović]

Pádraig Brady <P@draigBrady.com> writes:

> To get a better handle on the generality of this
> I went back about 16 years on the list,
> and even though there have been many tee functionality requests over the 
> years,
> this is the first time this has been requested.

I'm glad not to have wasted time on a redundancy!  I totally overlooked
checking the MLs, so I only searched through the "rejected features"
page; I'll remember to check archives too in the future.

> Trying to understand your use case better,
> I'm presuming the input generator is depending on the compiler runs
> (written to by tee) to exit cleanly, before exiting / generating more?
> Hence the hangs?

> If that was the case then there still might be the potential for hangs
> even if tee detected closed pipes. I.e. if the compiler runs hung rather
> than exited, this not be distinguishable from tee as the pipe outputs would 
> remain.

Yes, but these are fine.  The problem here isn't that the build process
can hang, the problem is that it can hang in a way different to the
compilers it's emulating.

The work tee is involved in consists of running the configure stage of a
package build against two compilers, universally, by essentially tee'ing
to both, storing and comparing the results, and then returning the
result of the "normal" compiler, in order to detect packages that
misconfigure due to compiler changes.

The bug we observed is that on occasion, for instance when running with
a tty, or with a script that (for some reason) has a pipe on stdin, the
tee-based "compiler" would hang.  To replicate this, try:

  tee >(gcc test.c -o a.out.1) >(gcc test.c -o a.out.2)

in a tty (here, the stdin is meant to be irrelevant).

In an applied version of this scenario, this would be cc-old/normal and
cc-new of some sort, through a wrapper function so that results are
stored/retrieved/compared/etc, but that complexity isn't necessary to
see to understand the issue I think (but [1] is the script involved in
the process, see the timeout line and the comment above it)

The behavior we'd want out of tee here is to let it pass down as much
data as possible to both compilers, and to be able to give up when the
last of the compilers die, hence waiting for POLLHUP events.

This is useful in other instances where stdin is not necessarily
relevant to what tee is sending data to, and where you can't deduce
whether it would be (at least, without fairly complex code to replicate
argument parsing).

> If that's the case this has become a bit less generally useful in my mind.
>
> To keep tee data driven, perhaps your input could periodically
> send a "clock" input through to (say a newline), to check everything
> is still running as expected.  I.e. your periodic input generator
> seems like it would be async already, so it would be better to
> add any extra async logic there, and keep tee more simple and data driven.

This is, sadly, not in our code.  We control (distro) build scripts
(ebuilds) and the compiler-diff-wrapper-thing, and the ability to run
(normal program) build systems unmodified, with just a custom CC, in
order to detect when they misbehave due to a compiler update is
paramount here.

> cheers,
> Pádraig.
>
> p.s. while reading the tee docs in this area, they were a bit dispersed,
> so the attached updates the texinfo for the -p option to summarise
> it's operation and how it differs from the default operation.

Thanks!  This seems nice.

Have a great night.

[1] https://gist.github.com/thesamesam/4ddaa95f3f42c2be312b676476cbf505
-- 
Arsen Arsenović

signature.asc
Description: PGP signature