[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Speeding up libtool

From: Robert Ögren
Subject: Re: Speeding up libtool
Date: Mon, 04 Apr 2005 01:30:26 +0200
User-agent: Mozilla Thunderbird 1.0 (Windows/20041206)

Hi Ralf,

Ralf Wildenhues wrote:
Hi Robert,

First I'd like to say many thanks for all your testing and valuable

Thank you for the responses.

Will take a little while to work through all of it.

Hehe, sorry, I had too much time on my hands last weekend and largely
spent it on fiddling with Libtool.

What it basically does is to build GLib 2.6.3 using different versions
of libtool and with and without caching. As I said before, I'm usually
using Cygwin but targetting MinGW (using gcc -mno-cygwin) to get native
win32 binaries, and pretending the host is also mingw to avoid having to
set up cross-compilation stuff. This works very well with one minor
change in Replace $LTCC $LTCFLAGS with /usr/bin/gcc at the
spot where it builds the wrapper exes from c source. (I don't expect you
to change this in Libtool, I'm just explaining how I use it. Maybe there is even a better way...)

It'd be interesting to know /why/ you have to change it, and what the
value of $LTCC and $LTCFLAGS are.

LTCC="gcc -mno-cygwin"
LTCFLAGS="-O1 -g -march=pentium -pipe -mms-bitfields -Wall"

The main reason I had to change it is that execv("/bin/sh",...) does not make sense in a native Windows program, which is what you get if the wrapper programs are compiled with gcc -mno-cygwin and a Cygwin environment is used. The Windows path to Cygwin's sh is something like C:\cygwin\bin\sh.exe, but this depends on where Cygwin is installed. Using plain cygwin gcc for compiling the wrappers solves this problem. There may be better solutions, I stopped searching when I found a simple workaround that worked for me.

I don't know how this works if MinGW and MSYS is used - I assume it must work somehow, otherwise people surely would have complained? I personally don't use MSYS, I find Cygwin much more complete and consistent, but that's just my personal opinion. Don't spend time on it if noone else complains.

The increase in time for the first compilation with Libtool HEAD is caused by a large amount of sed calls, mainly by the "Quote evaled strings" stuff from _LT_CONFIG_COMMANDS that is placed in config.status, since config.status is invoked a few extra times on win32 to create glib/glib.rc, gobject/gobject.rc, gmodule/gmodule.rc and gthread/gthread.rc. Each config.status invocation cost about 10 seconds. This is a minor thing, and could probably be fixed by adding them to AC_CONFIG_FILES in with some win32 conditional stuff.

Hmm.  It's possible to do a similar trick here as in general.m4sh, it's
just not clear how much it will gain.  And whether some old shell starts
screaming again.  Something along the lines of (this will be in

  var_xvalue=`eval $ECHO "X$var"`
  case $var_xvalue in
    eval "lt_\$var=..        # sed invocation
    eval "lt_$var=\$$var"

Attached is a lightly tested patch for this.  The speedup of
config.status execution would be interesting.  For small projects
this might be more pronounced.

With ash and lt_ECHO=printf...:
Without the patch:
sh ./config.status glib/glib.rc  takes approx 9.8 seconds
With the patch:
sh ./config.status glib/glib.rc  takes approx 8.8 seconds

I think the disappointing speedup is caused by a lot of spawning of printf (printf isn't a builtin in ash, right?) Setting ECHO=echo in config.status makes it take 5.7 seconds after the patch (8.2 before).

1. glib's defines a few variables like LT_RELEASE, and these names are forbidden with Libtool HEAD so I had to patch in a m4_pattern_allow for these

Yes.  I think it would be better if glib changed their names, at least
in the long run.  The autotools have more-or-less gone towards reserving
their respective prefixes AC_, AT_, AS_, AM_, LT_ for the respective tools.
It makes sense to forbid others, e.g. because that way typos can be
diagnosed.  For the time being, m4_pattern_allow is a decent workaround.


2. did not have AC_LIBTOOL_WIN32_DLL which now is required
for OBJDUMP to be set. However, even though I added AC_LIBTOOL_WIN32_DLL
before AM_PROG_LIBTOOL, it did not work, but if I replaced the call to
AM_PROG_LIBTOOL with LT_INIT([win32-dll]) it worked.

I think this is a bug in Libtool >= 2.0.  I'm not sure, though.

I sent a testcase in another mail. Looks like a bug to me.

3. (I had to modify libtool-cache so it removes the libtool: compile:
stuff that libtool HEAD prefixes output lines with)

Well, yeah, Libtool considered its output to be read by humans, not
parsed by more than our own testsuite.  It would, as already noted,
require very cautious audit for things like quoting level.

Agreed, I wasn't complaining :)

OTOH, if we can decide on putting a caching mechanism into libtool
proper, there might just be a simpler way than that.

Sure, but doing it in portable shell could be a challenge. For example, some recent tests show that all the stuff that is done by AS_SHELL_SANITIZE seems to be responsible for around 0.75 s execution time on Cygwin, due to a lot of subshell forking I think. For compilation mode, that is a significant part of the compilation time.

For Libtool HEAD I also rebuilt automake-2.59 and autoconf-1.9.2 so that
Cygwin's Libtool files would not be picked up.


- Using (a)sh instead of bash makes Libtool faster

ACK.  I would have thought the speedup was more than that, though.

I did a quick port of the test script to GNU/Linux and the speedup with ash was larger IIRC. If anyone is interested, I can provide more details.

- Libtool HEAD is faster than Libtool 1.5.10

I'm quite happy to see this is true (for more than the examples I

- libtool-cache still makes a lot of difference :)

Yes, quite true.  I am amazed at the amount of time spent in libtool. :(

Let's just say it was a large source of frustration.

- Cutting down the time for compile mode would help a lot since it is invoked so many times.

There is more than one side to this.  For one, the absolute times you
showed are much much larger for compilation than for linking.  OTOH,
usual developer cycle looks like:
  - edit a couple (1-3) files
  - compile these plus link
  - (maybe) test

and then it may make more sense to also look at how long that takes.

Yeah that is very true. I usually compile and work with larger things than glib, like GTK+ and GIMP, and there the time for linking clearly dominates in developer cycles like the above (without libtool-cache). In GIMP, a lot of the objects aren't even libtool objects.

Allow me to retract that conclusion. It is probably better to spend time on further improving linking if possible.

To get a rough idea of the relative speed: how many objects does glib

80 libtool objects and a few plain objects for testcases. 4 "real"
libraries, 2 convenience libraries, and 3 small test libraries as far as I can see. Plus 9 exes.

Other way round: how long to compile just a couple of objects?

Approximately 120 seconds to compile all 50 objects in the glib
subdirectory, with Libtool HEAD and (a)sh, i.e on average 2.4 seconds per object.

More conclusions:
- Improvements in cygwin/mingw fork() would be very noticeable.  :)
- configure is also quite slow (which becomes a pain if necessary to
run frequently).

This is probably something I should go ask the Cygwin developers about,
when/if I get some time.

My patch should help a tiny bit with this as well.

Probably a little yes, I haven't tested yet.

Again, thanks for the response


reply via email to

[Prev in Thread] Current Thread [Next in Thread]