Optimization for reading *.d files

[brenorg]

Hi everybody, 

I'm currently refactoring the build system at my company. We got around 20k
C++ source files, and we use a form of (messed up) recursive make. After a
lot of tinkering to get things right I got to 10 (maybe 5 with some more
massage) seconds for a clean build. I need something around 1 second. 

Anyway, I want to continue to use GNU Make, and not fallback to CMake/Ninja.
After some profiling, what's killing me is parsing the "*.d" files generated
by the compiler. 

The time to include all dependency files of my project in one single
makefile (as I want to get rid of recursive make), is 4 seconds. 

I decided to get my hands dirty. I cloned GNU Make 4.0 and started analyzing
the parser. Of course, the parser is complex. But those dependency files are
so so simple! I want to leverage that information. 

I added a detection so the parser knows it's parsing a ".d" file, and
switches to my own "eval_d_file" function. 

My function doesn't try to find special qualifiers, nor expand variables or
anything like that. It parsers as if the only allowed syntax is what exists
on what is generated by the compiler.

I got a speed-up. My original 4 seconds went to 2 seconds. Which awesome. Of
course, I probably did a buggy implementation (since I don't understand
everything going on the code), but it seemed to work. 

There are lots of dependency files and they can be processed in parallel,
before being merged into the database. 

For that, GNU make could need an extension on the include directive to
handle "include *.d" differently as it knows dependency files won't
alter/create variables but just add dependencies. 

I really don't have the guts to use my hackish make version in production,
but I'm up to develop this functionality, if you agree it's a good proposal. 

Thanks, 
Breno G. 





--
View this message in context: 
http://gnu-make.2324884.n4.nabble.com/Optimization-for-reading-d-files-tp17656.html
Sent from the Gnu - Make - Bugs mailing list archive at Nabble.com.