Re: speed up large library linking

[Ralf Wildenhues]

Hi Peter, everyone,

Picking up a thread that, incidentally, turns a year old this week,
about complexity reduction patches for link mode with many objects:
http://lists.gnu.org/archive/html/libtool-patches/2005-05/msg00158.html

There were justified concerns back then:

* Peter O'Gorman wrote on Fri, May 20, 2005 at 03:29:36PM CEST:
> Ralf Wildenhues wrote:
> >
> >How may I convince you?
> 
> I am not sure that you can convince me to like the temporary file usage, 

I won't, and don't have to, any more, at least for what I'm proposing
here.  Bash-3.1 has `var+=value' (like Zsh), and it's efficient, too.
:-))

So let's use it if available.

> and the use of utilitles like paste and join which are neither mentioned in 
> the GNU coding standards document or autocnf's portable shell section. The 
> fact that the files stick around if the user interrupts the process only 
> makes mattters worse. These patches are very destabilizing and, in my 
> opinion, make libtool harder to maintain.

Here's the rewrite of one of those 5 pending patches.
I think neither of your concerns apply to it any more.
I furthermore think that it is good to apply right now.

OK?

Some numbers:

Linking libcln from http://www.ginac.de/CLN/, 850 objects,
(the absolute values don't tell much, as they include linker
and archiver times; look at the relative differences only;
warm cache, minimum of several runs):

bash-3.1, HEAD and fast func_append:
3.32user 1.20system 0:04.57elapsed 98%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+69063minor)pagefaults 0swaps

bash-3.1, HEAD and slow func_append:
4.08user 1.22system 0:05.34elapsed 99%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+70255minor)pagefaults 0swaps

bash-3.00.14(1) and slow func_append:
3.95user 1.20system 0:05.18elapsed 99%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+67568minor)pagefaults 0swaps

bash-3.1, HEAD before func_append:
6.12user 1.29system 0:07.46elapsed 99%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+82143minor)pagefaults 0swaps

bash-3.00.14(1), HEAD before func_append:
5.95user 1.22system 0:07.21elapsed 99%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+67550minor)pagefaults 0swaps

bash-3.00.14(1), branch-1-5:
4.68user 7.49system 0:11.68elapsed 99%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+1007389minor)pagefaults 0swaps

Interestingly, even using the slow func_append, with a Bash that is
widely used at the moment, has a tremendous effect on performance.
Zsh wins a bit with this, too, but I think it may behave sanely even
with the previous version.

Note: the libtool.m4 part of the patch uses M4 quoting:
   eval "$[1]=\$$[1]\$[2]"

rather than alternate shell syntax:
   eval "${1}=\$${1}\${2}"

to inhibit M4 active symbols, because for the shell, the former makes
for a small but measurable improvement in a tight loop.

Cheers,
Ralf

        Eliminate quadratic scaling in argument parsing loop, for
        shells that understand `var+=value', such as Bash-3.1, or Zsh.

        * libltdl/m4/libtool.m4 (_LT_CHECK_XSI_SHELL): Test whether
        the shell understands `var+=value'.
        (_LT_PROG_XSI_SHELLFNS): Define `func_append' based on this.
        * config/ltmain.m4sh (func_mode_link): Use `func_append' for
        compile_command, finalize_command, libtool_args, libobjs,
        non_pic_objects.

speedup-parseargs5.diff
Description: Text document