Re: Speeding up libtool

[Ralf Wildenhues]

[ I don't have much time, so only this for now: ]

* Robert Ögren wrote on Mon, Apr 04, 2005 at 01:30:26AM CEST:
> Ralf Wildenhues wrote:
> >
> >It'd be interesting to know /why/ you have to change it, and what the
> >value of $LTCC and $LTCFLAGS are.
> 
> LTCC="gcc -mno-cygwin"
> LTCFLAGS="-O1 -g -march=pentium -pipe -mms-bitfields -Wall"
> 
> The main reason I had to change it is that execv("/bin/sh",...) does not 
> make sense in a native Windows program, which is what you get if the 
> wrapper programs are compiled with gcc -mno-cygwin and a Cygwin 
> environment is used. The Windows path to Cygwin's sh is something like 
> C:\cygwin\bin\sh.exe, but this depends on where Cygwin is installed. 
> Using plain cygwin gcc for compiling the wrappers solves this problem. 
> There may be better solutions, I stopped searching when I found a simple 
> workaround that worked for me.

I think we have avoiding the wrapper and using just a C program on the
TODO list.  That should also solve this problem along the way, I guess.
Alternatively, we'd need a way to get at the Windows path of sh.exe.
Wouldn't that be `cygpath -w /bin/sh'?

> >>The increase in time for the first compilation with Libtool HEAD is 
> >>caused by a large amount of sed calls, mainly by the "Quote evaled 
> >>strings" stuff from _LT_CONFIG_COMMANDS that is placed in config.status, 
> >>since config.status is invoked a few extra times on win32 to create 
> >>glib/glib.rc, gobject/gobject.rc, gmodule/gmodule.rc and 
> >>gthread/gthread.rc. Each config.status invocation cost about 10 seconds. 
> >>This is a minor thing, and could probably be fixed by adding them to 
> >>AC_CONFIG_FILES in configure.in with some win32 conditional stuff.

> >Attached is a lightly tested patch for this.  The speedup of
> >config.status execution would be interesting.  For small projects
> >this might be more pronounced.
> 
> With ash and lt_ECHO=printf...:
> Without the patch:
> sh ./config.status glib/glib.rc  takes approx 9.8 seconds
> With the patch:
> sh ./config.status glib/glib.rc  takes approx 8.8 seconds
> 
> I think the disappointing speedup is caused by a lot of spawning of 
> printf (printf isn't a builtin in ash, right?) Setting ECHO=echo in 
> config.status makes it take 5.7 seconds after the patch (8.2 before).

ash's `echo' is bad for libtool, as it interprets backslashes by
default.  The ash I use on linux has printf builtin.  I looked on
cygwin.com, they seem to ship an older ash variant without printf
builtin.  You may want to try a newer one or even try to get them to
include it in cygwin proper.  That would be a boon. :)

[ a caching mechanism ]
> >OTOH, if we can decide on putting a caching mechanism into libtool
> >proper, there might just be a simpler way than that.
> 
> Sure, but doing it in portable shell could be a challenge. For example, 
> some recent tests show that all the stuff that is done by 
> AS_SHELL_SANITIZE seems to be responsible for around 0.75 s execution 
> time on Cygwin, due to a lot of subshell forking I think. For 
> compilation mode, that is a significant part of the compilation time.

That really sucks.  Wonder who's the main culprit here..

> >>Conclusions:
> >>- Using (a)sh instead of bash makes Libtool faster
> >
> >ACK.  I would have thought the speedup was more than that, though.
> 
> I did a quick port of the test script to GNU/Linux and the speedup with 
> ash was larger IIRC. If anyone is interested, I can provide more details.

Because the ash you used there has builtin printf?

> >>- libtool-cache still makes a lot of difference :)
> >
> >Yes, quite true.  I am amazed at the amount of time spent in libtool. :(
> 
> Let's just say it was a large source of frustration.

I can understand.

> >>- Cutting down the time for compile mode would help a lot since it is 
> >>invoked so many times.
> >
> >There is more than one side to this.  For one, the absolute times you
> >showed are much much larger for compilation than for linking.  OTOH,
> >usual developer cycle looks like:
> >  - edit a couple (1-3) files
> >  - compile these plus link
> >  - (maybe) test
> >
> >and then it may make more sense to also look at how long that takes.
> 
> Yeah that is very true. I usually compile and work with larger things 
> than glib, like GTK+ and GIMP, and there the time for linking clearly 
> dominates in developer cycles like the above (without libtool-cache). In 
> GIMP, a lot of the objects aren't even libtool objects.
> 
> Allow me to retract that conclusion. It is probably better to spend time 
> on further improving linking if possible.

Both will be necessary.  Maybe we *should* re-think that ltmain.c idea...

[ snipped a lot of nice explanations ]

Thanks for your detailed response,
Ralf