make 3.82 performing more stat() calls than 3.81

[Troy Runkel]

Greetings,

 

The company I work for has a large code base consisting of many components.  As part of the build process we compute the dependency graph for these components and load it into GNU make.  We’ve noticed that processing these dependencies can take a significant amount of time in some cases.  For instance, on a Windows 7 build machine it takes over 30 minutes.

 

During our investigation we found that make 3.82 performs more stat() calls than 3.81.  In our Windows build environment we’re using a SMB2 based filer and these additional stat() calls result in network traffic to the filer.  We’re unsure why the stat() calls aren’t being satisfied by the local file cache, but that’s a separate issue.

 

Here’s a simple example of what we’re seeing.  Consider the following test Makefile.

 

build : $(addsuffix .ts,a b c)

 

%.ts :

touch $@

 

a.ts :

 

b.ts : a.ts

 

c.ts : b.ts a.ts

 

I build the Makefile the first time to create the *.ts files.  After that I use strace to execute make 3.81 on Linux.  The *.ts file already exist so there should be nothing to do.  Here are the stat() calls for my targets.

 

stat("a.ts", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0

stat("b.ts", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0

stat("c.ts", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0

 

For this test make 3.81 performs 1 stat() call per target file.  Next I use strace to execute make 3.82 on Linux.  Again, the *.ts files already exist.  Here are the stat() calls for my targets.

 

stat("./a.ts", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0

stat("./b.ts", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0

stat("./c.ts", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0

stat("./%.ts", 0x7fffc59db4c0)          = -1 ENOENT (No such file or directory)

stat("./a.ts", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0

stat("./b.ts", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0

stat("./a.ts", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0

stat("./c.ts", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0

stat("./b.ts", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0

stat("./a.ts", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0

stat("a.ts", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0

stat("b.ts", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0

stat("c.ts", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0

 

For this test it appears that make 3.82 performs N stat() calls per file, where N is based on the number of times each file is mentioned as a target or a dependency.  In our system, that means 1 stat() versus thousands of stat() calls per file.

 

Is this difference in behavior between make 3.81 and 3.82 intentional?  If so, is there anything we could do to reduce the number of stat() calls being made?  The additional stat() calls don’t have much of an impact on our NFS based Linux build environment, but they do have a noticeable impact on our SMB2 based Windows build environment.

 

Thanks.

 

--Troy Runkel