Re: RFC: indirect deplibs v2

[Ross Boylan]

On Fri, 2004-12-03 at 06:33, Ralf Wildenhues wrote:
> I haven't come as far as I would have liked (rpath discussion is very
> incomplete still), and I won't be responding for the next couple of
> days, but here's my current version of the RFC (pasted info plus texinfo
> source attached) for you to tear apart.
> 
> Thanks for all your valuable input so far, more feedback greatly
> appreciated.
> 
> Regards,
> Ralf

First, it might be useful to state or restate the problem(s) to be
solved.  This document, and the preceding thread, seem to be mixing a
variety of much different issues.  I am new to this list, and to
libtool, so the problems may be well known, but it strikes me a review
of them might be helpful.

I note several issues:
1) dependencies introduced at the source code level (e.g., by a
library's header files)
2) dependencies at the object level
3) incompatibilities induced by changing versions of libraries
4) a desire to avoid unnecessary link/compiles on systems with capable
linkers.
5) problems induced by attempting to link to multiple versions of the
"same" library (possibly even bit-identical files that live on different
paths?).
6) the question of whether dependencies can be derived automatically.

The fact that "interface" can refer to a header file, an object file's
exposed functions (more generally, smbols), or a particular ABI (as in,
incompatibilities across versions of g++) further muddies the waters.

It may also be relevant to distinguish dependencies derived at the
library level from those derived at the function or symbol level.  Take
the case of the algorithms library that calls libm.  At the source
level, a program may not use the definition from libalgorithms that
calls libm.  At the object level, an object may make no calls into libm,
and so have no direct dependency, and it may invoke no members of
libalgorithm that in turn invoke members of libm, and so have no
indirect dependency.  In this latter scenario, whether the object
depends on libm depends on whether we are looking at the library level
or a finer-grained level.  I suspect that on some systems the program
would run without libm, and that others might require it anyway.

> 
> 10 Inter-library dependencies
> *****************************
> 
> First, we define a set of terms to use later.
> 
>    A program or library has a "direct base library", if it depends on
> some interface that base provides (see *Note Interfaces:: for a more
> thorough description).
>      If you use `cos' in your program or library, then `libm' is a direct
>      base library, and your program or library is directly derived from it.
>    The program or library depending on the base library will be called
> "derived program or library", or short "derived object".  (1)

I find the use of class inheritance terminology an interference, rather
than a help.  First, class inheritance is generally acyclic, while
library relations are often cyclic.  Second, a base class may be direct
or indirect, at least in C++ useage (I think).  Third, why drag objects
and inheritance into the discussion at all?

I find the language of graphs and dependents more suitable.  The chronic
problem there is that sometimes the language is not clear about the
direction of the dependency.

I also suspect that the source code dependency issues are a red herring.

So, A depends directly on B if the object code in A references a symbol
exported from B.  A depends indirectly on B if A requires B in order to
run, but A does not depend directly on B.  (or perhaps use the language
of transitive closure of sets..)
> 
>    An object has an "indirect base library", if it does not depend on
> any interfaces of the library itself, but some base library of the
> object depends on such an interface.
This is not quite adequate, because it does not capture indirection that
occurs through several layers.  That is with the reading of "depend" as
"depend directly", which is implied by the first part of the sentence.
>      A program makes no use of any math routines itself, but uses a
>      library `libalgorithms', which uses the `cos' function.  Then
>      `libm' will be a indirect base library of the program, but a
>      direct library of `libalgorithms'.  Conversely, `libalgorithm'
>      is directly derived from `libm', but your program is indirectly
>      derived from `libm'.
> 
>    A library "exposes a base library" if it possibly causes a derived
> library or program to rely on an interface of that base.
What problem does introducing this concept help solve?
>      In example above, if `libalgorithms' is to be used with a header
>      file `algorithms.h', which contains
>        `#define mathop(angle) cos(angle)'
>      then the algorithms library exposes the math library.
>    In general, the notion of exposure is not detectable automatically.
> The following guide can be given:  Whenever library headers require
> exposing external interfaces to sources of derived objects, for example
> because they require inclusion of base library headers or include these
> headers themselves, the base library should be considered exposed to
> derived code.  These are not the only possibilities, however.
> 
>    A system with a "needed-following linker" has a means to record both
> dependencies on other libraries based on the soname of the base library,
yes, I think it would be good to define soname, and possibly use a
slightly more general term (esp for MS-Windows types).
Do you need the word "base" before "library" in the preceding quoted
sentence? 
> and rpaths in which those other libraries are to be searched.  It also
> has a linker that picks up both the soname as well as the rpath
> information to load all base libraries at runtime.  For example, the
> Linux linker does all of this.  (2) (3)
> 
>    Before Libtool version 2.2, the handling of inter-library
> dependencies has ignored the fact that some system linkers are smart
> enough to figure out the base libraries of base libraries (recursively)
> themselves, and always linked in all base libraries into the output.
> 
>    Whenever a lowlevel library changes its interface (soname), this
> requires recompiling of all derived objects.  
Do you mean "requires" in the sense of "the current design of libtool
requires it"?  Isn't one of the motivations for this effort that not all
systems actually do require a complete rebuild?

There are other issues:
* a change in the semantics of functions without a change in the
interface (probably don't need to rebuild callers)
* addition of new interfaces (under some circumstances, one can get away
without revising the callers, I think)
* changes in interfaces requiring a rebuild/link
> If most of the derived
> objects do not make use of the lowlevel interfaces in any way, this is
> suboptimal on systems with a needed-following linker.  It should
> suffice to recompile the directly derived objects (which itself do not
> change their interface).  On systems without needed-following linker,
> recompilation of all derived objects is necessary.
> 
>    In order to allow for portable linking using libtool, there are a
> few flags which can be added to the link line in order to specify the
> dependency information: The available options are `-exposed-libs',
> `-private-libs', `-direct-libs' and `-indirect-libs'.  These options
> behave like two types of switches and hold for the base libraries
> listed to the right of the option, until encounter of another switch of
> the same type.
At this point, I'm pretty lost about what those switches are supposed to
do.
> 
>    Libtool and calculates from the information given on the command
> line together with the information gathered from base libraries two
"base library" = indirect or direct base, in your terminology?
How about "required libraries"?
> sets:  A set of libraries to be directly linked to, and a set of
> libraries which are considered exposed to derived objects.  These sets
> are then recorded in the output.  (4)
> 
>    Speaking in terms of the directed, generally not acyclic, graph
> underlying inter-library dependencies: library dependency corresponds
> to reachability along a path, direct dependency corresponds to a path
> of length one, and exposure is an edge attribute.  The notion of a
Though I thought I was reasonably familiar with network terms, "edge
attribute" has me stumped.
Note that a library may be reachable through multiple paths, perhaps a
path of length 1 (direct) and length >1 (indirect, unless you adopt the
definition I proposed in which indirect = total set of required
libraries less the direct ones).
> "valid graph" then imposes certain connectivity conditions when
> inserting new vertices (which corresponds to creating derived object),
> namely:  directed paths of length two require a shortcut if the second
> edge of the path has the exposure attribute.  (5)
> 
>    The semantics are as follows:
> 
>    * If none of the options are specified on the command line, all base
>      libraries are considered direct and exposed.  Similarly, all bases
>      picked up from unadorned libtool libraries (i.e., the information
>      found in `.la' files lacking exposure or directness information)
>      are treated as direct, exposed base libraries.  (6)
> 
>    * The switches can be given as often as necessary and hold for the
>      following arguments (thus, order is important) until encounter of
>      another switch of the same type.  (7) Other than the necessity of
>      specifying libraries in order, the switches are pure logical
>      entities denoting edges and edge attributes to be added to an
>      existing graph.
> 
>    * Exposed bases of the object are always treated as direct bases,
>      whether so denoted or not.  The line `libtool --mode=link $CC -o
>      a.out libfoo.la -exposed-libs -indirect-libs libbar.la' will link
>      direcly against both `libfoo.la' as well as `libbar.la'.
> 
>    * An exposed base of a direct base library is a direct base, and
>      itself again exposed unless listed as private dependency.
> 
>    * If a library is mentioned (or picked up) as both direct and
>      indirect base, it is treated as direct base.  Direct bases of
>      direct bases are direct.
> 
>    * Otherwise, a base is indirect and not exposed.
I'm still a bit at sea.
> 
> 
>    On systems with needed-following linker, only the direct base
> libraries encoded in the output.  On all other systems (and of course
> when linking statically), all bases are encoded in the output.
> 
>    A few examples are probably necessary:
> 
>    TODO: Insert good examples here.
> 
>    TODO: RPATH discussion!
> 
>    Libtool < 2.2: Programs break if you install any newer library
> anywhere in the full rpath presented in program
> 
>    New semantics: Programs break if you install any newer or have a
> library with matching soname anywhere in the union of all picked-up
> rpaths and the set of all deplibs.
> 
>    TODO: Move the following to appropriate place.
> 
>    On systems without a needed-following linker, it is necessary to
> explicitly list all base libraries, direct as well as indirect.
> Libtool is smart enough to find all base libraries of libtool
> libraries.  But often, base libraries are not produces by libtool on
> such systems.  So portable linking needs to specify these indirect base
> libraries.
In the preceding paragraph, is the baseness of a library relative to the
lib or executable libtool is producing?  In general, is that enough info
for libtool?  Some (most?) linkers required an ordering of libraries
that depends on the interrelationships among the libraries, not just the
relations of the libraries to the main thing being built.

That's my last comment.  Thanks for your work on this; it's a hard
problem--probably a hard set of problems.

> 
>    On systems with needed-following linker, users are frequently
> tempted to not include indirect base libraries in the link line at all.
> Be warned, however, that this is not portable, because it will
> generally fail on other systems.  Only if your direct base libraries
> are guaranteed to be libtool libraries (`.la' files), e.g. because they
> are built as part of your software package as well, does omitting
> indirect base libraries work portably.
> 
>      libltdl.la depends on libdlloader.la,
>      libdlloader.la may depend on libdld, depending on the system.
>      libltdl.la depends on libdlloader.la,
>      libdlloader.la may depend on libdld, depending on the system.
> 
>    Since both `libltdl.la' and `libdlloader.la' are built within the
> same package and thus guaranteed to both be libtool libraries, it is ok
> to omit `-ldld' from the link line of `libltdl.la'.
> 
>    Open questions:
>    * Is it possible to find out in all cases which things actually mean
>      the same library?
> 
>    * Is it necessary for the previous question to be answered yes?
>      Also on systems without needed-following linker?
> 
>    * Is this really backwards-compatible?  I think so, libtool-1.5 and
>      newer seem to ignore `-indirect-libs' and `-direct-libs'.
> 
>    * Can it be implemented with reasonable effort?
> 
>    * What about rpath?
> 
>    ---------- Footnotes ----------
> 
>    (1) This discussion is not about legal aspects, and as such, we do
> not use this notion with the meaning it has with respect to law.
> 
>    (2) prominent counter-example
> 
>    (3) Do I have to define soname?  Should I use a different word?
> 
>    (4) If the output is either a library itself or loadable, see (*note
> Dlopened modules::).
> 
>    (5) This also means that libtool will possibly have to iterate over
> the set of libraries to consider until a stable set of new attributed
> edges is reached.
> 
>    (6) This is a safe, backward-compatible assumption, but the notion of
> full exposure might be too pessimistic, at least for system libraries.
> 
>    (7) One may restate the first point as implicitly added
> `-exposed-libs -direct-libs' at the beginning of the link line.
> 
> 
> ______________________________________________________________________
> _______________________________________________
> Libtool mailing list
> address@hidden
> http://lists.gnu.org/mailman/listinfo/libtool
-- 
Ross Boylan                                      wk:  (415) 502-4031
530 Parnassus Avenue (Library) rm 115-4          address@hidden
Dept of Epidemiology and Biostatistics           fax: (415) 476-9856
University of California, San Francisco
San Francisco, CA 94143-0840                     hm:  (415) 550-1062