Re: Library dependency files

[Todd Showalter]

On Fri, Apr 30, 2010 at 9:59 AM, Erik Rull <address@hidden> wrote:

>> @makedepend $(CFLAGS) $(SRCS)
>
> Okay - I didn't use that tool at the moment, but I've tested it immediately
> - looks great - except the cpp - headers are not found correctly, but the
> compiler does that afterwards (maybe with lost of time?).

    You have to hand all your include paths to makedepend for it to
behave properly.  A better option might be to instead do:

@gcc $(CFLAGS) -MM $(SRCS)

    The -MM tells it to ignore system headers when calculating
dependencies, which is usually a safe bet.  It would also mean that
there would be a lot less searching.  There's also a -MG option that
assumes missing headers are generated; that would probably also speed
dependency calculation up significantly, and you'd still catch any
actual missing headers at compile time.

> Yeah just finding out the dependencies is pretty fast - faster than the
> creation of the *.d files.

    That was how it seemed when I tried the per-file dependency route.

>>    Each directory has a file called "files.make", which contains a
>> list of the source files that are valid for that directory.
>
> Great - is this file created automatically or manually?

    Manually, because we wanted to be able to have files in the
directory without requiring they be built.  That way, incoming code
can live in-tree without having to be built.

> Very cool - I've tested some optimizations from my side here - the
> preconditions that are solved via makedepend improve the real compilation
> process only a little bit because of the issue with the c++ include files (I
> must include all default compiler search paths to get this working
> correctly), the time improvement is ~ 15% - 1:30 against 1:50 on a complete
> rebuild (Better than nothing :-)). The major time enhancement is the
> makedepend against the *.d file creation, the compiling process itself is
> not really faster.

    Probably the biggest thing there is the caching; if you can invoke
makedepend (or gcc, I believe) with all the files at once, it does a
lot of work to cache the results of its header parsing.  If a source
file asks to include a file makedepend (or gcc) has already seen this
invocation, it goes to the cache instead of wasting time re-reading
and re-parsing the file.  As a result, in a typical project where most
of the files include most of the same headers, your dependency
generation time for the whole project is only going to be marginally
more than for a single file, as long as you can do it all in one
invocation of the dependency generator.

    If you have to run it multiple times, though, obviously it forgets
everything it knows between invocations.

> I strace'd the make process and the compiler itself searches around a lot
> which takes a lot of time too.

> If you know further build improvement ideas - please let me know them.

    Well, definitely try the -MM -MG thing with gcc; I haven't tried
it myself yet (too many other things to do), but in principle it ought
to speed things up.

    There's probably not much to be done about the compile time
itself, unless you can switch to pure C.  If you've got any
shellscript wrapping the compile you might try losing it; it makes no
real difference on Linux, but in cygwin it can cause significant
delays in my experience.

    You might try distcc and ccache, though.

                                                                      Todd.

-- 
 Todd Showalter, President,
 Electron Jump Games, Inc.