Re: ports/94826: [patch] Very slow startup for libtool ltdl clients (eg gnucash)

[Ralf Wildenhues]

* Peter Jeremy wrote on Wed, Mar 22, 2006 at 07:42:33PM CET:
> I should acknowledge that the problem and fix were pointed out by
> address@hidden

Thanks, I'll mention Joerg.

> On Wed, 2006-Mar-22 17:28:11 +0100, Ralf Wildenhues wrote:
> >I'll put adding such a test on the TODO list.  I'll also put reworking
> >the manual loading algorithm on the TODO list, it shouldn't scale
> >nonlinearly in the number of loaded deplibs, however many modules we
> >open.
> 
> I've still got a trace of all calls to lt_dlopenext() during gnucash
> startup but I've cleaned up the more detailed debugging output that
> I collected (though I could probably re-create it).
> 
> I suspect it's a combination of:
> - both dlopen() and lt_dlopen() are recursively attempting to load
>   dependencies

Right.

> - The .la file can contain duplicate dependencies (which don't appear
>   to be stripped out)

This is "normal": there are systems where you actually need to link each
library against all its deplibs.  So we store all of them.

The linker and runtime linker on FreeBSD both follow DT_NEEDED entries
and look in DT_RPATH entries inside libraries for finding and loading
indirect library dependencies, though (I think).  So it should be
possible to improve quite a bit upon your 10 seconds startup time, by
changing Libtool to not link in indirect deplibs on FreeBSD (and
DragonFly).

This would be similar to the changes that Debian currently has in their
Libtool package.  We plan to fix need_all_deplibs=no (it will have a
different name then, and it has a couple of known issues at the moment),
and then make it available for all systems that support this.
(You'd have to relink all libraries once for this, though.)

> - It looks like duplicate entries in the library search path
>   (inherited from LD_LIBRARY_PATH and added via .la) aren't deleted

Ah, ok.  This is another limitation in libltdl.

> - Tracking lt_dlopen/lt_dlclose shows that libraries are being opened
>   and closed multiple times during startup.

Hmm.  Surely for FreeBSD this should be a non-issue now; but for systems
where we _do_ need to load all deplibs, I'm not sure whether we can get
around unloading them in the case that the module itself could not be
opened.  After all, those deplibs could have been the wrong ones to use
in the first place, but we're not able to know until we've tried loading
the module itself.

Luckily this issue is important on very few systems only, today.

> Before joerg pointed me in the right direction, I was looking at
> adding caching to lt_dlopenext() or lt_dlopen() but hadn't worked out
> how to correctly track reference counts across open/close calls.

I think that should be possible, but then again the details may depend
on the system in question.  I'll try to test on Tru64 when I have some
more time at hand.

Cheers,
Ralf