Re: Reducing dependency_libs to what was on the command line.

[Ralf Wildenhues]

Hi Kurt,

* Kurt Roeckx wrote on Mon, Dec 05, 2005 at 07:19:16PM CET:
> On Mon, Dec 05, 2005 at 09:33:54AM +0100, Ralf Wildenhues wrote:
> > 
> > When I have time, I'd like to fix your current issues with it,
> > namely cross-compilation[1], and wrong-libs-for-uninstalled-programs[2].
> 
> I'm very aware that it has some problems.  And hope to be able to
> work them out in the future.  I still believe this is the
> right way to go forward.

I sure agree that linking against all deplibs is a distribution
nightmare.

> I do not want to have a libtool in Debian that breaks even more,
> which is why I'm seeking input on this.

Good.

> > And also, I very very much would like to change the way this is enabled,
> > in order to both
> > - make this usable for non-Debianers
> > - not unconditionally turn this on for unsuspecting Debian users
> 
> I'm willing to do this, but I'd rather have that libtool does
> "the right thing" by default.  I know you have simular plans for
> a future release, and I hope this can be implemented soon.

Yes, me too.  Just that "the right thing" is a bit of a POV thing.

> > > And now I'd like to reduce it even more.
> > 
> > > What an other problem is that in case of convience libraries, it
> > > also adds all the libraries in the dependency_libs to the
> > > depends, and so links with all of those.  It should however only
> > > link with those they tried to link the convience library with,
> > > and not all it's dependency.
> > 
> > I don't understand.  Please post an example that exposes this change.
> 
> This is mostly a request of the Debian kde people, but in
> general for packages making use of conviences libraries.  Kde makes
> alot of use conviences libraries.  I was looking at the apollon
> package, but there are various others.  Apollon creates 1
> conviences library libkirc.la.  It does the following to
> create it:
> /bin/sh ../../libtool --mode=link --tag=CXX g++  [...]
> -o libkirc.la [...] dcchandler.lo kirc.lo kircmessage.lo
> -L/usr/X11R6/lib -L/usr/share/qt3/lib -L/usr/lib   -lkio

Ah, ok.  I didn't know people frequently put dependency information in
convenience archives.  Interesting.  Does it reflect the actual use of
symbols well?  I mean, do the objects in libkirc use symbols from libkio
only, but none of the deplibs of libkio:

> /usr/lib/libkio.la has this in it:
> dependency_libs=' -L/usr/lib -L/usr/share/qt3/lib -L/usr/X11R6/lib 
> /usr/lib/libkdeui.la /usr/lib/libkdesu.la /usr/lib/libkwalletclient.la 
> /usr/lib/libkdecore.la /usr/lib/libDCOP.la -lresolv -lutil 
> /usr/lib/libart_lgpl_2.la /usr/lib/libidn.la /usr/lib/libkdefx.la 
> /usr/lib/libqt-mt.la -laudio -lXt -ljpeg -lXi -lXrender -lXrandr -lXcursor 
> -lXinerama -lXft -lfreetype -lfontconfig -ldl -lpng -lXext -lSM -lICE 
> -lpthread /usr/lib/libXrender.la -lX11 -lz /usr/lib/libfam.la -lrpcsvc'


> And so libkirc.la ends up with:
> dependency_libs=' -L/usr/X11R6/lib -L/usr/share/qt3/lib -L/usr/lib 
> /usr/lib/libkio.la /usr/lib/libkdeui.la /usr/lib/libkdesu.la 
> /usr/lib/libkwalletclient.la /usr/lib/libkdecore.la /usr/lib/libDCOP.la 
> -lresolv -lutil /usr/lib/libart_lgpl_2.la /usr/lib/libidn.la 
> /usr/lib/libkdefx.la /usr/lib/libqt-mt.la -laudio -lXt -ljpeg -lXi -lXrender 
> -lXrandr -lXcursor -lXinerama -lXft -lfreetype -lfontconfig -ldl -lpng -lXext 
> -lSM -lICE -lpthread /usr/lib/libXrender.la -lX11 -lz /usr/lib/libfam.la 
> -lrpcsvc'
> 
> What I want is that it says:
> dependency_libs=' -L/usr/X11R6/lib -L/usr/share/qt3/lib -L/usr/lib 
> /usr/lib/libkio.la'
> 
> In case of convience libraries, it will add everything
> from the dependency_libs in the .la file to the depends,
> and not dependency_libs, so even with the Debian patch, it
> will link to all those libraries, which is what we want to
> avoid.

Ah, ok.  Well, again I believe people expect libtool to do the former,
i.e., add all of the stuff.  Again I think it would be useful to only
add the explicitly listed stuff, *given* that the author knows about
this.

Hmm.  For convenience archives from within the same software package,
this would not be so hard to communicate.  Installing convenience
archives is a stupid idea -- at least the .la file -- whoops.
Hey, there's a documentation bug: it still encourages to install
convenience archives like this:

  libtool --mode=install ./install-sh -c libhello.a \
                                        /local/lib/libhello.a

which will certainly not work (unless you pick the correct source
$objdir/libhello.a).

In any case, we don't have to worry about dependency information of
installed convenience archives.  Which is GOOD, as this means we don't
need to invent a new variable in the .la file (stating that the author
knows about the new convention of explicitly specifying all used libs).

> What my change does is basicly make it only add the -lkio
> to the depends.  I hope, but did not test, that everything
> in the libkio.la file ends up in the dependency_libs so
> that it doesn't break on other platforms.

Well.  Given above thoughts, I would think that your change makes sense
in the link_all_deplibs=no setting, given that the package author knows
about this.

Erm.  Except: what about static linking in that case?
I think there is an implicit assumption in ltmain that if
  P -> libA -> libB -> libC

then picking up libA's dependency_libs will give you all you need,
ie., you do not need to walk recursively.  For static linking, you will
need to do this now.  ltmain does not implement this walk at the moment.

> With my change, and the other Debian patch to not link to
> dependency_libs, this greatly reduces the number of
> libraries that gets linked too that it doesn't need itself.

Again, the very easiest way to find that out would be to write tests
for each of the above questions.  The new CVS HEAD Autotest test suite
may have a bit steep learning curve, but is really nice to use
afterwards.

> > > An other problem we've expierenced in the past is that in case we
> > > remove some lib in the chain, all libraries that got (inderectly)
> > > linked to it needed to be rebuild to remove the library from their
> > > dependency_libs in their .la files.  Having all the libraries you
> > > need to link to in the .la file is really not a good idea in my
> > > opinion.  I think this should work recursivly, and find the other
> > > libraries it should link to from the .la files it references.
> > 
> > Huh?  If you use symbols from it, you should link against it --
> [...]
> > Maybe I am misunderstanding you again.
> 
> Let's try to give an example of this.
> 
> Maybe I should give just start with the opposite example,
> since it's probably easier the follow.
> 
> I have a libA that links to a libB.  So libA's
> dependency_libs has libB in it.
> 
> Then I have a libC, which links against libA, but does not
> care at all about libB.  libC's dependency_libs will have
> libA and libB in it.
> 
> Now I start linking libA against libD too.  So libA's
> dependency_libs will have libB and libD in it, but libC's
> didn't change.
> 
> Now I try to link to libC, and if I only look at libC's
> .la file, it says it should link to libA and libB, but not
> to libD.  This mean that in case of static linking it will
> break, unless it also goes and looks at libA's .la file.

        P --> libC --> libA --> libB
                            `-> liBD

Correct.  It's an instance of the 'ltmain needs to do the walk'
issue.

> If I then link libC against libA again, it will be fixed,
> and libC's dependency_libs will list libA, libB and libD.
> 
> But then I decide that libA shouldn't link to libD after
> all, so libA's dependency_libs get updated, but libC's
> not.  This mean I should go and relink libC to libA to
> make that dependency go away.

Ah, ok!  Another class of problems I haven't thought of yet.

> libC should not need to get relinked in either case of
> adding or removing a dependency.  It's not required for
> either static or dynamic libraries.

True.

> What I would like to see is that libC's dependency_libs
> only have libA in it, and should never contain libB or
> libD, which is just an "implementation detail of libA".

True, in the link_all_deplibs=no case.

> libtool should just go look at libA's .la file if it needs
> to know what the current dependencies for libA are.  And I
> think it already does that.
> 
> So the 2 problems I've mentioned (convience libs and
> adding/removing dependencies) both seem to have a simular
> solution, reducing the dependency_libs to what was on the
> command line when making that library.

Well, yes, but solving this without breakage simply requires that
ltmain walk the dependency chain.

> > > So, I've tried what the effect is of having only things you link
> > > to in the dependency_libs of the .la file.  And it already seems
> > > to be working recursivly, so I have to wonder why they're all
> > > there.  Are there any plans on changing that behaviour soon?
> > 
> > Again, please show with an example of how it behaves before/after the
> > patch.
> 
> I hope the above examples make this clear now.  Please ask
> more questions if it's not clear. 

Much better.

> > What systems have you tested on?

> I only have access to GNU/Linux, so I can't test this
> myself.

If you can help out providing tests, I can easily test them on a number
of platforms

> > > Do you think this can break something?
> > 
> > Yes.  Very certainly the patch below can and will break something.
> > I mean, it's even unconditionally for any architecture.  No Go.
> 
> I hope this doesn't break anything.  The only thing I
> currently see how it can break is that it maybe doesn't
> add dependency_libs where it should.

Well.  If people link convenience archives directly in programs
(which they shouldn't, but they do), then this patch will break it.

You also need to take into account that quite a few users
- either don't install the .la files (which we may rightfully,
  declare as "it's their problem then", but in reality that will
  be another maintenance nightmare)
- or, and that is more important, where a lot of the installed
  libraries simply were never created with libtool, e.g., because
  they stem from a proprietary OS.  There, having indirect dependency
  information is very valuable.

> I could make this architecture specific, but I think we
> should be able to solve those problem so that it doesn't
> break.

It should be at least specific to link_all_deplibs=no.
And it should pass all tests outlined above.

> Note that I have no intention of uploading this to Debian
> as long as I'm not sure it doesn't break anything for
> other architectures.

Good.

Cheers,
Ralf