Re: Add support for limiting CPU pressure

[Randy MacLeod]

On 2022-12-21 11:24, Paul Smith wrote:
> On Tue, 2022-12-20 at 16:59 -0500, Randy MacLeod wrote:
> > > While we are doing development with Yocto Project (1), a source-
> > > based Linux distro builder, we found that many instances of making
> > > running in parallel with other work can overload even a large
> > > many-core build machine. Existing mechanisms that are either not
> > > system-wide (-j) or are too slow(-l has 1 minute averaging), so in
> > > order to make optimal use of a multi-core system for some larger
> > > tasks we need a new mechanism.
> Can you clarify what version of GNU make you're using?
4.3 from ubuntu-22.04 but the build pool for Yocto
has ~15 distros of various vintages as well as being
able to use the version of make (4.4) that we carry on

the oe-core master branch:

  https://layers.openembedded.org/layerindex/recipe/245/
http://cgit.openembedded.org/openembedded-core/tree/meta/recipes-devtools/make/make_4.4.bb

For any work we do here, we'd use that master branch version rather than
the older host version.

> There have been a number of efforts to improve the accuracy of -l and
> it does not (any longer) use a simple load detection, for EXACTLY this
> reason (the value is too coarse, particularly at startup).
Okay. I'll try to allocate time to see what happens with 4.4.
> In the latest version GNU make 4.4, it will use the content of
> /proc/loadavg (if it exists) to figure out how many jobs are currently
> executing on the system and use that as a measurement of load.
>
> If that's not available, we use getloadavg() but we do some math on it
> to modify the load average based on the number of jobs we've started in
> the last second.  Of course, this also suffers from the problem that it
> only knows about jobs that THIS INSTANCE of make has started in the
> last second.
>
> Anyway you can read more about this in the comments in the source code:
>
> https://git.savannah.gnu.org/cgit/make.git/tree/src/job.c?h=4.4#n1953

Thanks, I missed the recent changes amongst the decades old code! ;-)

What tests were you running to check these improvements?

--
# Randy MacLeod
# Wind River Linux