Re: Speeding up libtool

[Robert Ögren]

Hi Ralf,

Ralf Wildenhues wrote:
> Hi Robert,
>
> First I'd like to say many thanks for all your testing and valuable
> feedback. 

Thank you for the responses.

> Will take a little while to work through all of it.

Hehe, sorry, I had too much time on my hands last weekend and largely
spent it on fiddling with Libtool.

...
> > What it basically does is to build GLib 2.6.3 using different versions
> > of libtool and with and without caching. As I said before, I'm usually
> > using Cygwin but targetting MinGW (using gcc -mno-cygwin) to get native
> > win32 binaries, and pretending the host is also mingw to avoid having to
> > set up cross-compilation stuff. This works very well with one minor
> > change in ltmain.sh: Replace $LTCC $LTCFLAGS with /usr/bin/gcc at the
> > spot where it builds the wrapper exes from c source. (I don't expect you
> > to change this in Libtool, I'm just explaining how I use it. Maybe there 
> > is even a better way...)
>
>
> It'd be interesting to know /why/ you have to change it, and what the
> value of $LTCC and $LTCFLAGS are.

LTCC="gcc -mno-cygwin"
LTCFLAGS="-O1 -g -march=pentium -pipe -mms-bitfields -Wall"

The main reason I had to change it is that execv("/bin/sh",...) does not 
make sense in a native Windows program, which is what you get if the 
wrapper programs are compiled with gcc -mno-cygwin and a Cygwin 
environment is used. The Windows path to Cygwin's sh is something like 
C:\cygwin\bin\sh.exe, but this depends on where Cygwin is installed. 
Using plain cygwin gcc for compiling the wrappers solves this problem. 
There may be better solutions, I stopped searching when I found a simple 
workaround that worked for me.

I don't know how this works if MinGW and MSYS is used - I assume it must 
work somehow, otherwise people surely would have complained?
I personally don't use MSYS, I find Cygwin much more complete and 
consistent, but that's just my personal opinion. Don't spend time on it 
if noone else complains.

...
> > The increase in time for the first compilation with Libtool HEAD is 
> > caused by a large amount of sed calls, mainly by the "Quote evaled 
> > strings" stuff from _LT_CONFIG_COMMANDS that is placed in config.status, 
> > since config.status is invoked a few extra times on win32 to create 
> > glib/glib.rc, gobject/gobject.rc, gmodule/gmodule.rc and 
> > gthread/gthread.rc. Each config.status invocation cost about 10 seconds. 
> > This is a minor thing, and could probably be fixed by adding them to 
> > AC_CONFIG_FILES in configure.in with some win32 conditional stuff.
>
>
> Hmm.  It's possible to do a similar trick here as in general.m4sh, it's
> just not clear how much it will gain.  And whether some old shell starts
> screaming again.  Something along the lines of (this will be in
> config.status)
>
>   var_xvalue=`eval $ECHO "X$var"`
>   case $var_xvalue in
>   *[\\\`\"\$]*)
>     eval "lt_\$var=..        # sed invocation
>     ;;
>   *)
>     eval "lt_$var=\$$var"
>     ;;
>   esac
>
> Attached is a lightly tested patch for this.  The speedup of
> config.status execution would be interesting.  For small projects
> this might be more pronounced.

With ash and lt_ECHO=printf...:
Without the patch:
sh ./config.status glib/glib.rc  takes approx 9.8 seconds
With the patch:
sh ./config.status glib/glib.rc  takes approx 8.8 seconds

I think the disappointing speedup is caused by a lot of spawning of 
printf (printf isn't a builtin in ash, right?) Setting ECHO=echo in 
config.status makes it take 5.7 seconds after the patch (8.2 before).

..
> > 1. glib's configure.in defines a few variables like LT_RELEASE, and 
> > these names are forbidden with Libtool HEAD so I had to patch in a 
> > m4_pattern_allow for these
>
>
> Yes.  I think it would be better if glib changed their names, at least
> in the long run.  The autotools have more-or-less gone towards reserving
> their respective prefixes AC_, AT_, AS_, AM_, LT_ for the respective tools.
> It makes sense to forbid others, e.g. because that way typos can be
> diagnosed.  For the time being, m4_pattern_allow is a decent workaround.

Ok

> > 2. configure.in did not have AC_LIBTOOL_WIN32_DLL which now is required
> > for OBJDUMP to be set. However, even though I added AC_LIBTOOL_WIN32_DLL
> > before AM_PROG_LIBTOOL, it did not work, but if I replaced the call to
> > AM_PROG_LIBTOOL with LT_INIT([win32-dll]) it worked.
>
>
> I think this is a bug in Libtool >= 2.0.  I'm not sure, though.

I sent a testcase in another mail. Looks like a bug to me.

> > 3. (I had to modify libtool-cache so it removes the libtool: compile:
> > stuff that libtool HEAD prefixes output lines with)
>
> Well, yeah, Libtool considered its output to be read by humans, not
> parsed by more than our own testsuite.  It would, as already noted,
> require very cautious audit for things like quoting level.

Agreed, I wasn't complaining :)

> OTOH, if we can decide on putting a caching mechanism into libtool
> proper, there might just be a simpler way than that.

Sure, but doing it in portable shell could be a challenge. For example, 
some recent tests show that all the stuff that is done by 
AS_SHELL_SANITIZE seems to be responsible for around 0.75 s execution 
time on Cygwin, due to a lot of subshell forking I think. For 
compilation mode, that is a significant part of the compilation time.

> > For Libtool HEAD I also rebuilt automake-2.59 and autoconf-1.9.2 so that
> > Cygwin's Libtool files would not be picked up.
>
> OK.
>
> > Conclusions:
> > - Using (a)sh instead of bash makes Libtool faster
>
> ACK.  I would have thought the speedup was more than that, though.

I did a quick port of the test script to GNU/Linux and the speedup with 
ash was larger IIRC. If anyone is interested, I can provide more details.

> > - Libtool HEAD is faster than Libtool 1.5.10
>
> I'm quite happy to see this is true (for more than the examples I
> tested).
>
> > - libtool-cache still makes a lot of difference :)
>
> Yes, quite true.  I am amazed at the amount of time spent in libtool. :(

Let's just say it was a large source of frustration.

> > - Cutting down the time for compile mode would help a lot since it is 
> > invoked so many times.
>
> There is more than one side to this.  For one, the absolute times you
> showed are much much larger for compilation than for linking.  OTOH,
> usual developer cycle looks like:
>   - edit a couple (1-3) files
>   - compile these plus link
>   - (maybe) test
>
> and then it may make more sense to also look at how long that takes.

Yeah that is very true. I usually compile and work with larger things 
than glib, like GTK+ and GIMP, and there the time for linking clearly 
dominates in developer cycles like the above (without libtool-cache). In 
GIMP, a lot of the objects aren't even libtool objects.

Allow me to retract that conclusion. It is probably better to spend time 
on further improving linking if possible.

> To get a rough idea of the relative speed: how many objects does glib
> have? 

80 libtool objects and a few plain objects for testcases. 4 "real"
libraries, 2 convenience libraries, and 3 small test libraries as far as 
I can see. Plus 9 exes.

> Other way round: how long to compile just a couple of objects?

Approximately 120 seconds to compile all 50 objects in the glib
subdirectory, with Libtool HEAD and (a)sh, i.e on average 2.4 seconds 
per object.

> More conclusions:
> - Improvements in cygwin/mingw fork() would be very noticeable.  :)
> - configure is also quite slow (which becomes a pain if necessary to
>   run frequently). 

This is probably something I should go ask the Cygwin developers about,
when/if I get some time.

> My patch should help a tiny bit with this as well.

Probably a little yes, I haven't tested yet.

Again, thanks for the response

Robert