[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Speeding up libtool

From: Robert Ögren
Subject: Re: Speeding up libtool
Date: Tue, 15 Mar 2005 20:59:39 +0100
User-agent: Mozilla Thunderbird 1.0 (Windows/20041206)

Ralf Wildenhues wrote:
Hi Robert,

Hi Ralf,

For the HEAD branch of Libtool, I have made some changes that cause it
to fork less often.  For the particular bad case I was looking at
(linking against ~500 objects), I got it from about 11 sec to about 5
sec, both of which include 2.5 sec of time spent actually linking (all
data cached) on a fast linux/x86_64 machine.

The improvements also benefit the general case, but are most likely less
pronounced, even if Cygwin should be the candidate to benefit most.

If you have resources/time to test, it would be nice to see a comparison
for your projects.  For best results on Cygwin, you should probably
configure (with libtool HEAD) like
  CONFIG_SHELL=/bin/ash lt_ECHO='printf %s' /bin/ash configure [...]

Interesting, I'll have to find some time to try that soon.

By replacing the name of the source file and the corresponding output file with placeholders if they match it usually only requires a few libtool invocations for compiling many source files. The link commands are usually unique but at least the second time a source tree is rebuilt they are already cached.

Ah, ok.  For use within the same $buildtree and compilation only this
should be mostly fine (save locking, as mentioned in your README).

Linking is a problem, though: shell wrappers break, as do generating
symbol lists.  Can't think of more right away, but I'm sure there are
many more corner cases which break.

Shell wrappers? If you mean the shell scripts that are created to set up PATH etc. before running an uninstalled EXE from .libs, they are also saved by libtool-cache by using the same mechanism that is used to save the .la{,i} files (serialize_ltfile) and it seems to work.

I just open the generated files, transform them into a set of echo statements that recreate the files, and splice them into the command sequence that Libtool outputs. The corresponding wrapper .exe file that is generated by Libtool when running on Cygwin is just copied from a precompiled version.

I don't handle the generation of symbol lists apart from ignoring the line that Libtool prints when it's generating it, so that is probably broken. I haven't needed it yet...

How much is the actual speedup in numbers?

I will have to get back to you after doing some proper benchmarks, but you can find at least some numbers here:

This is on a quite fast machine with lots of RAM for disk cache, however I did leave the antivirus software running (although set to ignore the source tree). I should turn that off and see if it makes any difference.

> How much with the HEAD branch of Libtool?

Sorry, I must admit I haven't actually tried it yet, although I did check out the source from anoncvs recently. I'll try it soon.

Iff this gives a decent speedup, I think the idea
could belong in Libtool proper.  That is, if you and the other Libtool
developers are so inclined.

I certainly wouldn't mind this functionality being included in Libtool (or perhaps just becoming unnecessary if Libtool is rewritten in C or bytecode). If anyone wants to they should feel free to improve my crude code or just take the useful ideas and reimplement them properly. I can't promise that I'll have time myself to do that implementation.

Thinking out loud more: if this were created as output of `configure'
(possibly acting upon some configure and/or libtoolize switch), it would
also be possible to make libtool-cache (more) portable and much more
foolproof.  For example, if configure decides locking is necessary, the
cache could be bypassed.  $objdir, PATH_SEPARATOR, etc. would be
trivially available, the `libtoolcacheize' beast could be killed..

That would be great.

We could add a marker whenever libtool does non-cacheable things so that
the cache would be bypassed then.  Or just let libtool output
"everything" it does so that even writing wrappers would work.
Maybe into a separate file descriptor.

Yes, that would be nice. Right now I remove some messages that Libtool writes, and try to detect warning messages and skip caching in such cases. But it is not perfect.

Most likely the cache would have to be invalidated whenever
config.status changes.  And it would have to be $top_builddir-local, so
you'd have several caches for several subpackages.

But for all of this to work reasonably, I think libtool needs to be
audited so it outputs the executed commands very closely.


I'd also think it would be possible to rewrite libtool-cache in portable
shell, but then remaining speedup would have to be evaluated (one `perl'
exec is, after all, not cheap).  I'm pretty sure this would be more
useful as a separate script than being directly integrated into the
`libtool' script, since many shells need a long time for the initial
parsing of `libtool'.

Also agree.

Minor nit for your current script: Why do you remove comments from the
created .la files?  I don't see a benefit here.

That is just to reduce the size of the cache for speed. I don't actually generate the .la files and wrapper script, I just save the ones that Libtool generates, which gives a rather long command line to save, even without comments.

I use an embarassingly simple "database" format - just a flat file where each line contains first the Libtool command line, a separator, and then the corresponding set of commands to be executed. Then I just read the file sequentially until the desired command is found or EOF occurs. (There are actually three cache files, one for compiling, one for linking libs and one for linking exes.)

That was just the first try, and it worked well enough so I didn't bother using a better database format. I'm sure there are Perl modules that can help with this.

Note that it's probably not worth investing work unless there is a
demonstrable speedup involved.

I completely agree. I'll try to make some better test runs and post the results here soon. I'll try to script the tests as far as possible to make it easier for other people to try and reproduce the results. It would also be useful to confirm that there indeed exist people who need a faster Libtool, to justify the added complexity of caching. My current code works fine for my modest needs, so don't spend time on it unless you think it will benefit other people.

To be honest, so far I only know of one person besides me that actually have used libtool-cache (hi Tor :). There might be others, but they haven't told me and SourceForge's statistics system is currently broken. I haven't really promoted the code either, didn't feel it was sufficiently tested.


reply via email to

[Prev in Thread] Current Thread [Next in Thread]