[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Make, MAKE_TERMOUT, color escape sequences, TTYs, and PTYs

From: Brian Vandenberg
Subject: Re: Make, MAKE_TERMOUT, color escape sequences, TTYs, and PTYs
Date: Wed, 1 Jun 2016 08:38:56 -0600

On the one hand I can see some value in this effort, but it seems simpler for build maintainers to overtly request colorized output if that's what they want.  If the tools start generating escape sequences for other than colorizing text it may not be sufficient, but in general it seems like a better strategy.

However, if you're going to go down this path ... it would be extremely helpful if there was a built-in mechanism to automatically send output to individual logs; adding that functionality as part of this effort seems like it wouldn't be a terribly difficult thing to do.

I've recently made changes to our build to get colorized output not just in the terminal but in logs as well, and added an option to get rid of colorized output if they don't want it.  At the heart of what I'm doing:

* Using .ONESHELL with bash
* At the beginning of every recipe is stuff for logging that dup()s stdout/stderr so I can print log paths to the terminal for recipes that fail (using a trap in bash), then either redirects output to logs or tees to both the logs & terminal.
* Added a check to see if a variable (eg, "NO_COLOR") is set or not to decide whether to pass flags to clang/gcc to colorize output

For developers like me who are more likely to "cat" the logs this works wonderfully, with the caveat that greping is a pain because of the escape sequences in the logs, but that's a hindrance I'm willing to put up with in exchange for the other conveniences.


On Wed, Jun 1, 2016 at 7:56 AM, Paul Smith <address@hidden> wrote:
On Tue, 2016-05-31 at 15:06 -0700, Josh Triplett wrote:
> However, if programs start observing those variables, that seems
> highly likely to lead to potential breakage in makefiles, for one key
> reason: those variables remain in the environment even for programs
> run with stdout/stderr redirected or piped.

I agree that this is a complication.  My thought here was that these
variables would only be monitored by a few specific tools where it makes
sense, not by every tool which might possibly want to know whether
output was a TTY or not.

For example, I would think that GCC/binutils/clang/cmake/etc. would
definitely WANT to consult these variables, so that colorized warnings,
etc. could be emitted.

On the other hand, I would expect that simpler tools like ls, grep, etc.
that might generate colorized output would NOT want to consult these
variables, because the likelihood that they are being used in some other
context where the variables are not accurate (and where the risk of
doing the wrong thing) is very high.

In general I anticipated that use of these variables would be restricted
to development toolchain programs where it's highly likely that the
variable values are correct and not vestigial, and in the rare
situations where they are not people can unset them in the recipe or use
explicit tool options that disable colorized/etc. output (which is
probably safer in those situations where the output is postprocessed

> In order to collect output via a pty, make will need to continuously
> read data from the pty and write it out to the temporary file.  I was
> thinking of doing so either by forking an extra process to do the
> copying when launching a command, or by having the parent process do
> this for all commands it runs as part of its pselect loop.

This is the main reason why I didn't want to mess with ptys (in addition
to general portability): not being able to use simple redirection is a
big complication.

I am not excited about the idea of adding a read/write loop into the
pselect loop.  I think this will really slow down the build if programs
make is invoking are forced to wait because their output is hung writing
to the pipe, until make gets around to servicing input on all the pipes.
 People are using GNU make on systems with 32, 64, even more CPU cores
with high -j values.

On the other hand, it's not going to work to have make fork a copy of
itself just to manage those pipes.  Make processes can get very large
for today's large, non-recursive builds.  Even with COW fork
capabilities, each forked instance of make itself will be a significant
overhead.  I don't like that idea.

On the gripping hand, make has always been a trivial installation
operation because it's just one executable which is completely
relocatable.  Having it build a little extra program to do data
shoveling from the pipe then installing that, having make find it, etc.
is a big extra complication.

Of course, a simpler solution would just be for make to invoke cat(1) or
something to handle this instead of providing its own program.  Possibly
we'd need to allow the makefile author to reset this variable to some
other tool so the name should be in a make variable.  I don't know if
the overhead of this will be an issue: will having 64 instances of
cat(1) running with your 64 instances of gcc mean you can't use as high
a -j because of the extra processing needed?  Cat is a simple tool, but
it's not zero cost like redirection.

I feel like a non-empty set of people would rather use IO redirection,
than pay a penalty of an extra forked program just to deal with
colorized tool output.

Bug-make mailing list

reply via email to

[Prev in Thread] Current Thread [Next in Thread]