[RFC] New library "type" needed?

[Charles Wilson]

The problem:
=======================================================================
Currently, libtool supports several types of libraries (I'm glossing 
over some stuff here, be gentle):

(1) Normal shared libraries
(2) Normal static libraries
(3) Convenience libraries
    (a) non-PIC -- will eventually be "exploded" and subsumed
        into a normal static library, or used by an in-project
        executable
    (b) PIC -- will eventually be included into a normal shared
        library via '--whole-archive libfoo --no-whole-archive'
        or something similar; or perhaps included in a PIE
        executable(?).
(4) Modules
    (a) shared
    (b) static, dlpreopen

As far as I can tell, none of these types satisfy the following need -- 
and perhaps this "need" is win32-specific; I'm not sure.

I have a library that I want to build shared (let's call it "libbfd"). 
It depends on a portability library that is currently built as a 
non-libtool, static library (let's call it "libiberty"). <g>

Now, it's bad to included non-PIC objects in a shared library (defer 
discussion of the meaning of "PIC" on win32 'til later, but consider the 
patch under consideration to add "real" -fpic support:
[PATCH,i386][4.3][RFC] PIC Generation on windows/cygwin
http://gcc.gnu.org/ml/gcc-patches/2007-02/msg00855.html

So, mindful of this, on non-win32 platforms the libiberty machinery 
manually builds a static archive of PIC .o's and a different static 
archive of non-PIC .o's.  I modified the build machinery to do this also 
on win32.  However, libtool can't tell the difference -- and, 
_currently_ on win32 there really is no difference -- so it rejects 
building libbfd shared, because "you can't build a shared library that 
depends on a static library".

So, next, I dug up an old (Oct 2004) patch that built libiberty using 
libtool.  That patch went thru several iterations trying to deal with 
multilib support that was lacking in the ancient libtool, before finally 
being withdrawn.  Since I am currently experimenting with Steve Elcey's 
recent work
Updating libtool in GCC and srctree
http://gcc.gnu.org/ml/gcc/2007-03/msg00293.html
bringing in modern libtool, I expect those old difficulties may no 
longer apply...but that's an issue for later.  One other thing the 
2004/10 patch did was AFTER building with libtool, it removed the actual 
libtool libs and created its own .a's using the libtool-built .o's.  I 
didn't keep THAT part of the old patch; I wanted actual libtool libraries.

However, this led to another problem: depending on whether I 
libtool-link using -rpath or not, I get either:

(1) two convenience libs, on with "PIC" .o's and the other with 
"non-PIC" .o's, or
(2) an actual shared cygiberty-0.dll and a "normal" static libiberty.a

In the first case, here's what happens when I try to build libbfd with 
-liberty using libtool

(1a) the static libbfd gets all of the objects of the static libiberty.a 
convenience lib included within it.  This is bad because I really don't 
want libbfd to have those symbols.

(1b) the shared libbfd ALSO gets all of the objects from the shared 
libiberty.a convenience lib, and auto-exports all of those symbols. 
This is also bad -- I want to use libiberty to satisfy internal 
undefined symbols within libbfd, but do not want to export those symbols 
OUTSIDE of libbfd.  I tried adding -Wl,--exclude-libs -Wl,libiberty.a to 
libbfd_LDFLAGS, and that actually worked...but
    (i)   it is not at all portable
    (ii)  requires me to know that the convenience library name is
          exactly "libiberty.a", rather than relying on libtool
          knowing how to translate "-liberty" -> "libiberty.la" ->
          "(whatever)"
    (iii) confusing: the actual link command includes both
          -Wl,--exclude-libs -Wl,libiberty.a AND
          -Wl,--whole-archive libiberty.a -Wl,--no-whole-archive
          So I'm still including all of the objects in libiberty within
          libbfd, even though I'm not exporting any of the libiberty
          symbols.  Libiberty might have 50 objects, but libbfd may only
          need two of them -- lotsa bloat.

(2a) This works as you would normally expect; the libiberty _LIBADD is 
dropped from the 'ar' command when building the libbfd static library.

(2b) This also works, but is problematic.  The interface of the shared 
libiberty library is not stable. It depends on what objects are deemed 
necessary for a given target (or host, or even build, sometimes) and 
multilib setting.  Normally, that's the time you should use -release 
MY_PACKAGE_VERSION instead of -version-info MY_C:MY_R:MY_A -- but 
libiberty is shared at a *source* level between multiple projects, which 
may be built differently, at different times. Besides, even if the build 
variant issue wasn't a problem, libiberty doesn't even HAVE release 
numbers we could use, so we'd need to come up with something to use...

This is also an issue for gnulib.

I considered maybe some sort of hash of the basenames of the .o's 
included in the archive, plus the canonical platform name for the 
in-process compilation (which, in the context of the overall 
binutils/gcc build, might be 'host' or 'target' or 'build'), plus the 
multilib settings, to generate some funky -release value:

cygiberty-27c14b30d89d2bb565643edc2fd6ef25.dll

But that'd be really fragile, especially on multilib. And it's ugly. And 
I never want to see 163 different *iberty*.dll files in my /bin.


Brainstorming:
=======================================================================
What I really want is a type of convenience library (call it a 
"resolver" library?) that

(1) is built both PIC and non-PIC, depending of course on the value of 
enable_shared and enable_static.  So far, just like a "normal" 
convenience lib.

(2) where the non-PIC resolver library is ignored when building 
dependent static libraries (and dependent static [dlpreopened] modules). 
 That is, NOT exploded and included within the dependent.  However, the 
non-PIC resolver library should be used like a normal convenience 
library when building dependent, in-package executables that depend on 
it directly (worry about PIE here? We don't worry about it at present 
anywhere that I can see).

(3) where the PIC resolver library is used *like a static library* when 
building dependent shared libraries -- that is, used to satisfy 
undefined symbols in the shared library if -no-undefined, but where the 
objects in the PIC resolver library are included wholesale via 
--whole-archive/--no-whole-archive -- and better yet, on win32 they 
should be excluded from auto-export using -Wl,--exclude-lib -Wl,<PIC 
resolver library name>.  Also, these "static" libraries should not 
trigger the "you can't create shared libs with static dependencies" 
filter within libtool.  (Hmm. When -no-undefined is NOT specified, then 
there's a choice: you could satisfy as many undefined symbols as you 
could by passing the resolver lib to the linker, or you could simply 
drop the resolver entirely -- which would match the current build 
procedure for libbfd & friends on non-win32...)

I don't think it would be that difficult to add this facility to libtool 
-- another command line argument for link mode, re-use most but not all 
of the code to handle convenience libraries, and (unfortunately) add 
another keyword to the .la format.  But I'd rather not -- especially 
this close (?) to 2.0final -- unless there were no other way to avoid 
the difficulties above.  OTOH, since it is likely that ToT libtool will 
be going in to binutils/gcc/newlib/etc sooner rather than later, I'd 
really like to be able to build shared stuff in those trees for a LOT of 
reasons -- not least, the ability to get a working, modern, g++ on win32 
(google 'dll exceptions gcc')!

Comments, discussions, or better ideas?

--
Chuck