Re: Library dependency files

[Erik Rull]

Hi Todd,

Todd Showalter wrote:
> On Thu, Apr 29, 2010 at 1:57 PM, Erik Rull <address@hidden> wrote:
>
> > Very interesting - how does your make rule look like for this case?
>
> @makedepend $(CFLAGS) $(SRCS)

Okay - I didn't use that tool at the moment, but I've tested it immediately 
- looks great - except the cpp - headers are not found correctly, but the 
compiler does that afterwards (maybe with lost of time?).

>     The single invocation of makedepend with all sources allows it to
> cache aggressively.
>
> > My "doing nothing" takes currently ~ 4 seconds - how large is your project?
>
>     ~200K lines of source across ~500 .c and .h files.  Any given
> build uses roughly 95% of the sources.

Yeah just finding out the dependencies is pretty fast - faster than the 
creation of the *.d files.

>     I think I've described the scheme here before, but the basic idea
> is that we have our source tree organized in a specific way:
>
> Kernel/
>    Graphics/
>       arch.OpenGL/
>       arch.NintendoWii/
>       arch.SonyPSP/
>       ...
>    ...
>
>     Each directory has a file called "files.make", which contains a
> list of the source files that are valid for that directory.

Great - is this file created automatically or manually?

>     We have a tool ("BuildList").  You tell it where the source tree
> is, and what architecture flags it should use.  It walks the source
> tree looking for files.make files, and builds a list of sources.  If
> (for example) we have a case like this:
>
> Math/Matrix44.c
> Math/arch.NintendoDS/Matrix44.c
>
>     and "NintendoDS" is one of the architecture flags set for this
> build, "Math/Matrix44.c" is discarded and
> "Math/arch.NintendoDS/Matrix44.c" is added to the list of sources.  If
> no architecture flags match the subdirectories, the file in the root
> is taken.  This lets us share most of the source code between our
> architectures without having to resort to #ifdef very often.
>
>     Once the source list is built, we hand that to makedepend to
> generate a dependency file, and append the source list to it.  Note
> that we don't call makedepend per-file, we call it once with the
> complete source list so it doesn't have to re-parse all the header
> files for each source.
>
>     At this point, we have a complete file containing (1) all the
> source files, (2) all the object files, which we've marked up to build
> in $(OUTDIR)/$(ARCH)/Objects, [1], and (3) all the dependency info.
> We recurse to another instance of make, and hand it the complete
> recipe along with the arch-specific build instructions.
>
>     So, makedepend gets invoked once with all the sources, and when
> make goes to actually build the project it has everything it needs in
> one file.  This has given us fairly quick builds, even on cygwin.[2]
>
>                                                                   Todd.

Very cool - I've tested some optimizations from my side here - the 
preconditions that are solved via makedepend improve the real compilation 
process only a little bit because of the issue with the c++ include files 
(I must include all default compiler search paths to get this working 
correctly), the time improvement is ~ 15% - 1:30 against 1:50 on a complete 
rebuild (Better than nothing :-)). The major time enhancement is the 
makedepend against the *.d file creation, the compiling process itself is 
not really faster.

I strace'd the make process and the compiler itself searches around a lot 
which takes a lot of time too.

If you know further build improvement ideas - please let me know them.

Best regards,

Erik