Re: Speeding up libtool

[Robert Ögren]

Ralf Wildenhues wrote:
> Hi Robert,

Hi Ralf,

> For the HEAD branch of Libtool, I have made some changes that cause it
> to fork less often.  For the particular bad case I was looking at
> (linking against ~500 objects), I got it from about 11 sec to about 5
> sec, both of which include 2.5 sec of time spent actually linking (all
> data cached) on a fast linux/x86_64 machine.
>
> The improvements also benefit the general case, but are most likely less
> pronounced, even if Cygwin should be the candidate to benefit most.
>
> If you have resources/time to test, it would be nice to see a comparison
> for your projects.  For best results on Cygwin, you should probably
> configure (with libtool HEAD) like
>   CONFIG_SHELL=/bin/ash lt_ECHO='printf %s' /bin/ash configure [...]

Interesting, I'll have to find some time to try that soon.

> > By replacing the name of the source file and the corresponding output 
> > file with placeholders if they match it usually only requires a few 
> > libtool invocations for compiling many source files. The link commands 
> > are usually unique but at least the second time a source tree is rebuilt 
> > they are already cached.
>
>
> Ah, ok.  For use within the same $buildtree and compilation only this
> should be mostly fine (save locking, as mentioned in your README).
>
> Linking is a problem, though: shell wrappers break, as do generating
> symbol lists.  Can't think of more right away, but I'm sure there are
> many more corner cases which break.

Shell wrappers? If you mean the shell scripts that are created to set up 
PATH etc. before running an uninstalled EXE from .libs, they are also 
saved by libtool-cache by using the same mechanism that is used to save 
the .la{,i} files (serialize_ltfile) and it seems to work.

I just open the generated files, transform them into a set of echo 
statements that recreate the files, and splice them into the command 
sequence that Libtool outputs. The corresponding wrapper .exe file that 
is generated by Libtool when running on Cygwin is just copied from a 
precompiled version.

I don't handle the generation of symbol lists apart from ignoring the 
line that Libtool prints when it's generating it, so that is probably 
broken. I haven't needed it yet...

> How much is the actual speedup in numbers? 

I will have to get back to you after doing some proper benchmarks, but 
you can find at least some numbers here:

http://mail.gnome.org/archives/gtk-devel-list/2005-March/msg00062.html

This is on a quite fast machine with lots of RAM for disk cache, however 
I did leave the antivirus software running (although set to ignore the 
source tree). I should turn that off and see if it makes any difference.

> How much with the HEAD branch of Libtool?

Sorry, I must admit I haven't actually tried it yet, although I did 
check out the source from anoncvs recently. I'll try it soon.

> Iff this gives a decent speedup, I think the idea
> could belong in Libtool proper.  That is, if you and the other Libtool
> developers are so inclined.

I certainly wouldn't mind this functionality being included in Libtool 
(or perhaps just becoming unnecessary if Libtool is rewritten in C or 
bytecode). If anyone wants to they should feel free to improve my crude 
code or just take the useful ideas and reimplement them properly. I 
can't promise that I'll have time myself to do that implementation.

> Thinking out loud more: if this were created as output of `configure'
> (possibly acting upon some configure and/or libtoolize switch), it would
> also be possible to make libtool-cache (more) portable and much more
> foolproof.  For example, if configure decides locking is necessary, the
> cache could be bypassed.  $objdir, PATH_SEPARATOR, etc. would be
> trivially available, the `libtoolcacheize' beast could be killed..

That would be great.

> We could add a marker whenever libtool does non-cacheable things so that
> the cache would be bypassed then.  Or just let libtool output
> "everything" it does so that even writing wrappers would work.
> Maybe into a separate file descriptor.

Yes, that would be nice. Right now I remove some messages that Libtool 
writes, and try to detect warning messages and skip caching in such 
cases. But it is not perfect.

> Most likely the cache would have to be invalidated whenever
> config.status changes.  And it would have to be $top_builddir-local, so
> you'd have several caches for several subpackages.
>
> But for all of this to work reasonably, I think libtool needs to be
> audited so it outputs the executed commands very closely.

Agreed.

> I'd also think it would be possible to rewrite libtool-cache in portable
> shell, but then remaining speedup would have to be evaluated (one `perl'
> exec is, after all, not cheap).  I'm pretty sure this would be more
> useful as a separate script than being directly integrated into the
> `libtool' script, since many shells need a long time for the initial
> parsing of `libtool'.

Also agree.

> Minor nit for your current script: Why do you remove comments from the
> created .la files?  I don't see a benefit here.

That is just to reduce the size of the cache for speed. I don't actually 
generate the .la files and wrapper script, I just save the ones that 
Libtool generates, which gives a rather long command line to save, even 
without comments.

I use an embarassingly simple "database" format - just a flat file where 
each line contains first the Libtool command line, a separator, and then 
the corresponding set of commands to be executed. Then I just read the 
file sequentially until the desired command is found or EOF occurs. 
(There are actually three cache files, one for compiling, one for 
linking libs and one for linking exes.)

That was just the first try, and it worked well enough so I didn't 
bother using a better database format. I'm sure there are Perl modules 
that can help with this.

> Note that it's probably not worth investing work unless there is a
> demonstrable speedup involved.

I completely agree. I'll try to make some better test runs and post the 
results here soon. I'll try to script the tests as far as possible to 
make it easier for other people to try and reproduce the results. It 
would also be useful to confirm that there indeed exist people who need 
a faster Libtool, to justify the added complexity of caching. My current 
code works fine for my modest needs, so don't spend time on it unless 
you think it will benefit other people.

To be honest, so far I only know of one person besides me that actually 
have used libtool-cache (hi Tor :). There might be others, but they 
haven't told me and SourceForge's statistics system is currently broken. 
I haven't really promoted the code either, didn't feel it was 
sufficiently tested.

Robert