|
From: | Eric Melski |
Subject: | Re: Prioritizing non-dependent targets in parallel make |
Date: | Mon, 04 Jan 2010 14:19:48 -0800 |
User-agent: | Thunderbird 2.0.0.23 (X11/20090817) |
Tim Murphy wrote:
Hi, I'm nobody official. I have seen Electric Accelerator using history files which may be doing something a bit like this. Personally I don't like the idea of priorities very much. Large builds that I have done do have big targets but since we are building 5000 of the same kind of target (using a macro to define a generic template and $eval to instantiate it for each specific target) it becomes impossible for us to provide hints. In the end we'd have to force the users of our build system say "webkit.dll" is pretty huge or "euser.dll" isn't. Since this is all relative information, how can we expect a build made up of components that various people produce to have the right "relative" weights? So I, personally, want something that learns. I would like to be able to store the learned information into source control and use it as the basis for the next attempt. One would have to check for patents on this, I think.
Hi Tim!ElectricAccelerator doesn't factor runtimes into scheduling decisions, although we have talked about doing so in the past. I spent some time investigating the possibility, most recently a couple years ago. What I did was tweak the Simulator report in ElectricInsight to simulate this kind of scheduling, so each time it chooses the next job to "run", it picks the longest job of those that are available to start. I ran the simulation on a variety of real-world builds.
Unfortunately, the results were disappointing. In my recollection, the best case gave a savings of about 5% of the total build time, and in general it was barely measurable -- maybe 1%. It seems that in practice, most of the jobs in a build are about the same length. The notable exception is the final link, which is of course much longer. But then -- of course -- that can't start until all the other jobs are done anyway, so the fancy scheduling doesn't help.
I'll see if I can dig up the patch for ElectricInsight and maybe write it up in a blog. up that code, and maybe write it up in a blog.
Cheers! Eric Melski Architect Electric Cloud, Inc. http://blog.electric-cloud.com/
[Prev in Thread] | Current Thread | [Next in Thread] |