bug-make
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [bug #40225] Deterministic output ordering


From: Tim Murphy
Subject: Re: [bug #40225] Deterministic output ordering
Date: Sat, 12 Oct 2013 13:28:35 +0100

> Me too. I think we all basically agree that all of the above can be
> useful, we just disagree on how hard each is to implement. I'm not
> so sure these "very few little enabling features" are really such
> small changes, whereas I think ordered output and/or individual log
> files are easier to implement. Of course, I may be biased since I
> was involved with the output-sync changes and don't know the other
> parts of make so well.

.ONESHELL working on windows is already almost enough.  It's possible
to do all of this stuff without it using macros and the '\' line
continuation character but makefiles start looking horrible and you
can't get the benefit on existing makefiles. It would be really
wonderful to have some way to trap 'the completion of a recipe',
whether or not that recipe was successful, and then execute something


One problem with reordering is how do you do it without actually
affecting the way you schedule jobs?

e.g. one makefile I have worked on a lot had 20,000 jobs that could
start immediately if there were 20,000 processors to run on.

If the order in which these tasks completed was random then
re-ordering them would require that one potentially buffer all job
output in memory.

So, you might say, it's not random because there are 64 CPUs and jobs
are allocated round-robin so there's some window of something around
64 (perhaps more) bits of output that one would have to keep given
that the scheduler is still iterating over jobs in the same order and
given that the execution time of each job doesn't change.

But now what if make decides to execute jobs in a different order from
one build to the next for some very good reason?  What if you run the
build on a machine with a much faster disc than CPU and jobs stop
completing in the same way.  That would make it a lot less scalable to
implement.

So one possibility would be to enable whatever output re-ordering data
is needed and pipe it through tools into diff.  This way you can
implement something that works in these smaller builds and people with
bigger problems can implement a different approach.

e.g. diff  oldbuild.log <(reorder.py newbuild.log)

-- 
You could help some brave and decent people to have access to
uncensored news by making a donation at:

http://www.thezimbabwean.co.uk/friends/



reply via email to

[Prev in Thread] Current Thread [Next in Thread]