For Your Consideration
======================
Hello,
this is an implementation of a new option for Make. It controls
the
number of parallel jobs in regards to the amount of memory in use
on a
host. It allows one to use Make's parallel job execution on hosts
with
multiple CPU cores, but where memory is limited. It works similar
to
the -l, --load-average option and has to be used together with the
-j,
--jobs option.
For example, assuming a host with 2GB of memory, the following
options
are equivalent:
-u25%, -u0.25, -u512m, or -u0.5g
In this example will Make stop creating new jobs when a host's
memory
use reaches 512MB (-j1), and will resume when it drops back below
the
threshold (-jN). A memory threshold can be specified as either an
absolute value or as a percentage of the host's total memory.
When the option is used in combination with -l, --load-average
then
Make stops creating new jobs when either the load threshold has
been
reached, or when the memory threshold has been reached. In
addition
will Make use the load and memory thresholds as a hint for a job's
average memory use and scales the number of jobs (from -j1 to
-jN).
This reduces load and memory spikes near the thresholds and
stabilizes
the memory usage of a Make run.
A value of 0 turns it off (-u0) and sets it to unlimited, which is
also the default.
The attached patch is against release 4.3 of Make. It implements
the
-u option for Linux using /proc/meminfo, and it enables the use of
/proc/loadavg for the -l option.
How well does it work?
======================
To show its effectiveness in practise did I create graphs for
system
loads (upper graphs) and memory usages (lower graphs) of different
Make runs. In these graphs is an average shown (thicker line)
together
with the raw values (thinner lines, spikes). Because the memory
use
can vary between runs did I animate the graphs by using 5 Make
runs.
The "std-" labels in the legends mean that the standard Make 4.3
release was used. The "new-" labels mean the patched version was
used.
The following is a Linux kernel build on a 16-core CPU with 32GB
of
memory. The first graphs (black) show standard runs with only the
-j
option. The second graphs (red) show standard runs with -j and -l.
The
third graphs (green) show runs with -j and -l, using /proc/loadavg
under Linux. The forth graphs (blue) show runs with -j, -l and -u
combined, using /proc/loadavg and /proc/meminfo. The additional
graphs
show variations of the arguments to the -u option:
https://zippyimage.com/images/2022/02/12/183964b428e4a0f0530c647c923924b9.gif
The same kernel build on a 4-core SoC with 512MB:
https://zippyimage.com/images/2022/02/12/aa05eb906c4bb1f6eb969b5bc29b7731.gif
To test the limits did I use a Makefile from image processing. It
consists of 3 rules and processes 30 files, leaving little room
for
Make to control its jobs. The jobs themselves use multiple
threads,
which inflate the system load, and use a fair amount of memory:
https://zippyimage.com/images/2022/02/12/189b22aa75adc0a6b7a34b6b9cceabee.gif
A more practical test is a build of LLVM/Clang-13 with itself,
using
link time optimization (LTO). The Makefiles are CMake-generated.
Here
a link job with LTO can use up to 24GB of memory. The host itself
has
got 32GB of RAM plus 16GB of ZRAM for swap, making it near
impossible
to build with a fixed number of parallel jobs. A "make -j8" will
consistently run out of memory and out of swap space (red,
failed). A
"make -j8 -u30%" will build it 5 times without failing. Other
values
for -u work, too, however only builds, which succeeded at least 5
times and not failed once, are shown in the graphs. The
combination of
-l with -u together works better and allows to build with "-j16
-l16
-u90%" in under an hour:
https://zippyimage.com/images/2022/02/12/9837a5551cc9350c17806c7b57efb991.gif
The last test is a LLVM/Clang-13 LTO build with "-j16 -l16 -u90%"
and
CCACHE enabled. It speeds up the build time and brings the LTO
link
jobs closer together, causing them to further overlap. It builds 5
times without failing:
https://zippyimage.com/images/2022/02/12/a55d2c3352a134edcb6af3feed8c2ff6.gif
The patch is provided "as is".
I hope you enjoy it and find it useful.
patch-memory-used