[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Undesirable overhead in GNU Make 4.2.1.

From: Hans Ottevanger
Subject: Undesirable overhead in GNU Make 4.2.1.
Date: Tue, 6 Feb 2018 15:12:45 +0100


We are attempting to use GNU Make 4.2.1 to build a large software tree
(~290000 targets).

The build hosts we are testing on are (virtualized) quad core X86-64
machines running RHEL 6. Our code tree is currently on "locally
perceived" storage. We compiled GNU Make 4.2.1 from sources.

Our main concern at this moment is the time needed for "nil
incrementals", i.e. the time needed to establish that nothing has
changed, so nothing needs to be done. If we put all dependencies in
the Makefile itself or in one single included makefile, we manage to
do a nil incremental in about 12 seconds.

But we plan to use auto-generated dependencies and thus include a .d
file in our main Makefile for each target. Currently auto dependency
generation is possible for about 90000 of our targets. Nil incremental
builds cost increases to about 140 seconds in that case, which is
quite unacceptable for our usage. We found that this overhead
increases approximately quadratically with the number of included
dependency files.

It appears that this behavior is caused by the need to maintain the
variable MAKEFILE_LIST. The names of all included makefiles are
appended one by one to the ever growing string representing this
variable.This also explains why memcpy() and strlen() champion the
profiled runs we did and also the quadratic behavior we observed. When
we change the source (read.c) so that the MAKEFILE_LIST is not
maintained at all, the overhead is gone and we are back to about 12
seconds again.

Are there any ideas (or opinions) about how to get rid of this type of
undesirable overhead? One of our own ideas is to provide a special
variant of -include (called e.g. dinclude?), intended to include just
.d files, without maintaining MAKEFILE_LIST. On the other hand it
might be feasible to improve the efficiency of appending to a
variable. Any suggestions?

By the way, Ninja needs about 7 seconds for a nil incremental with
about 90000 .d files.

Kind regards,

Hans Ottevanger

reply via email to

[Prev in Thread] Current Thread [Next in Thread]