[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [RFC] New library "type" needed?

From: Charles Wilson
Subject: Re: [RFC] New library "type" needed?
Date: Thu, 29 Mar 2007 01:03:41 -0500
User-agent: Thunderbird (Windows/20070221)

Ralf Wildenhues wrote:
Here's a review based on the question how such a concept could be
expressed portably.  I don't see how something like this would be of
great help if it could not be made to gracefully decay into something
still usable, on at least a useful set of platforms if not all of them.

Agreed. While I was considering the problem, I kept getting the feeling that either (1) I was missing something, or (2) this was very win32 specific: it's a problem only for platforms that require -no-undefined for shared libraries, which I think is only win32 and AIX.

* Charles Wilson wrote on Sun, Mar 25, 2007 at 09:34:47PM CEST:
What I really want is a type of convenience library (call it a "resolver" library?) that

(1) is built both PIC and non-PIC, depending of course on the value of enable_shared and enable_static. So far, just like a "normal" convenience lib.

Well, currently a convenience lib is not built in both ways at the same
time, but that's only a side issue for the discussion at hand.

Really? I thought I had both; perhaps it was just accumulated clutter from multiple builds with different settings. In any case, I can see where the current behavior you describe could be problematic on some platforms, including win32, in certain cases.

(2) where the non-PIC resolver library is ignored when building dependent static libraries (and dependent static [dlpreopened] modules). That is, NOT exploded and included within the dependent. However, the non-PIC resolver library should be used like a normal convenience library when building dependent, in-package executables that depend on it directly

OK so far.  You want a convenience archive without the whole-ness.
Should it be possible to install this thing or not?

Ummm...maybe? Let me ping back a few more questions: how would you install/distinguish between the two versions (no-pic and pic)? Would we need a naming convention of some sort and teach ld/gcc about it (a la' .dll.a)

This could get ugly.

On the other hand, currently binutils/gcc/etc install libiberty.a -- but do not install any headers for it. So the only was to use an *installed* libiberty is to declare the relevant functions yourself (or go dig around in the src for the headers), AND hope that whoever built the libiberty.a you want to link against, built it in such a way that it includes all the functions you want.

In short, *currently* libiberty is not well-handled. Since the intent for that library is to share at the source level, a better policy would be for binutils/gcc/etc to affirmatively change their build structure to NOT install libiberty.a.

And nobody talks about installing a libgnulib.a -- especially as gnulib-tool lets you change the library name...

So maybe the answer to the question SHOULD BE no, even if at present the actual practice IS to install libiberty.a.

Can it itself have
nontrivial (i.e., non-convenience) dependencies?

Errr...the two extant examples (libiberty and gnulib) do not, but...

The first of these questions is really a crux of the matter: if libbfd
can be shared and installable, then really I don't see any way except
- either explicitly forbid any third party to use your library, even
  though you install it shared,
- or have at least some sort of stable interface notion.

Yes. Since libiberty is explicitly unstable in its interface, I believe it really makes no sense to install it; it's just of no use outside the build tree.

Following this line of reasoning, I think my "resolver" library concept is really just a variant of a convenience lib -- in the sense that convenience libs are not usually (ever?) considered as installable objects.

(worry about PIE here? We don't worry about it at present anywhere that I can see).

We have very minimal PIE support only currently in CVS HEAD, in compile

Don't know enough about this issue; I'll let it pass...

(3) where the PIC resolver library is used *like a static library* when building dependent shared libraries -- that is, used to satisfy undefined symbols in the shared library if -no-undefined, but where the objects in the PIC resolver library are not included wholesale via

[ "not" correction included ]

--whole-archive/--no-whole-archive -- and better yet, on win32 they should be excluded from auto-export using -Wl,--exclude-lib -Wl,<PIC resolver library name>.

Lots of systems don't allow symbol hiding (or, at least, Libtool
currently doesn't support it well on most systems).  What should happen
for them?  More to the point: what if libbfd uses the resolver libiberty
release x, while libfoo used release y of libiberty, we're on a system
without symbol hiding, and my program links against both libbfd and
libfoo?  The order of linkage will determine in which library the blow
up occurs, be that silent or spectacular, depending on how incompatible
x and y are.

But on how many systems are BOTH of the following true:
  (1) to build shared requires -no-undefined
  (2) but symbol hiding is unavailable

You know, thinking about this more, I guess (1b) from my previous email is not THAT bad -- for shared libraries; just a little bloat of the final library -- it requires a little platform-specific gobbledy-gook in, but libbfd (and libobject, etc) already do that:

bfd/ from gcc trunk:
# Horrible hacks to build DLLs on Windows.
case "${host}" in
  if test "$enable_shared" = "yes"; then
WIN32LIBADD="-L`pwd`/../libiberty -liberty -L`pwd`/../intl -lintl -lcygwin -lkernel32"
  # We borrow WIN32LIBADD so that the shared libbfd won't depend on
  # libiberty.a.
  case "${host}" in
    # Linux/MIPS uses PIC by default.
    if test "$enable_shared" = "yes"; then
      WIN32LIBADD="-L../libiberty -liberty"
x=`sed -n -e 's/^[ ]*PICFLAG[ ]*=[ ]*//p' < ../libiberty/Makefile | sed -n '$p'`
    if test -n "$x"; then
      WIN32LIBADD="-L../libiberty/pic -liberty"

Now, there are things in the snippet above I'd do differently (like: no need for cygwin and kernel32; shouldn't just declare conditionals, and the actual additions to _LIBADD/_LDFLAGS be done in under the control of those conditionals; etc).

What's amusing though, is that what STARTED as a "horrible hack for win32" got hijacked not just by *-*-linux, but also by EVERY other platform in the default clause!

Anyway, this same machinery -- since it is there already -- can be used to insert -Wl,--exclude-lib in the right place, for win32.

I'd still like to be able to build my convenience library as both pic and non-pic tho. And I still want to prevent libiberty.a(non-pic) from getting the --whole-archive treatment when it comes to libbfd.a.

Hmmm.  More on that, below.

Also, these "static" libraries should not trigger the "you can't create shared libs with static dependencies" filter within libtool.

Several systems simply don't allow to mix PIC and non-PIC symbols.
On w32 this warning is pathetic, but on others we would have a problem
here.  (I can see why, on w32, you'd want to kill the warning here.)

No, you're missing the point of my statement: I *know* you don't want to mix pic and non-pic. In fact, I'm sorta kinka trying to make win32 behave that way, too, at least in this case, as an 'inoculate against the future' effort. (in case "true" -fpic support for win32 is merged into gcc trunk).

What I meant is, since in my hypothetical, we are being careful to have one .a full of pic symbols, and another .a full of non-pic symbols, then when we are creating a shared library and are explicitly using the .a full of pic symbols -- in THAT limited case, we should not get the FAILURE message about "you can't create shared libs with static dependencies"

This is not a warning, not even on win32. It's a failure; libtool dies, make reports error, bang you are dead. This is because we currently can't tell, just from looking at the .a, if the objects are pic or non-pic. It took a lot of dancing just to teach func_win32_libid() how to distinquish between import libraries and "regular" static libraries. Now, *if* there truly were a difference (on win32) between "pic" and "non-pic" objects, we'd have a fighting chance to educate func_win32_libid -- but at present, it's a distinction without a difference.

So, that's why I suspected that an implementation of my idea would require an additional keyword in the .la file -- so that libtool linkmode could tell (from the .la) that

"yes yes, I know file_magic_cmd reports that this is a "static library" and ordinarily that would be a problem, but since this particular dependency claims to be a "resolver", I know I can use the "pic" version (stored as the first element of library_names, where the .dll.a would usually go) without trouble. So no warn, no fail..."

(Hmm. When -no-undefined is NOT specified, then there's a choice: you could satisfy as many undefined symbols as you could by passing the resolver lib to the linker, or you could simply drop the resolver entirely -- which would match the current build procedure for libbfd & friends on non-win32...)

I don't think it would be that difficult to add this facility to libtool --

No, it wouldn't.  The semantics are the hard part.  If we can't get
well-defined semantics, the new baby will be useful for binutils (and of
course the set of packages that use it) only.

Possibly.  Also might be useful for packages that use gnulib.

But, it still smells to me like this whole problem might be a win32-only thing. Like THAT's new. So, since win32 has symbol hiding, maybe a little bloat is not that bad: build as a regular convenience archive, libtool will include it entirely into the eventual target lib via --whole-archive, BUT a little configury/automake magic in these limited cases to finagle --exclude-lib into _LDFLAGS.

(again: okay for shared lib. But static target lib...)

So, instead of new library "type", is there some sort of directive we could implement instead? Say, something like:

--------- --------

libtarget_LDFLAGS = -no-undefined -exclude-lib
libtarget_LIBADD =

Where the semantics of the libtool option '-exclude-lib' would be

(1) arguments must be convenience libraries (if functionality is useful, later modifications could relax this)

(2) libconvenience.a is given the --whole-archive/-no-whole-archive treatment when building the shared libtarget, but ONLY if the target architecture supports symbol hiding (see below). libconvenience.a is also marked with "-Wl,--exclude-libs -Wl,libtarget.a".

(2a) libconvenience.a is included in dependency calculations and everything for the static libtarget.a, but is NOT given "explode and incorporate into libtarget.a" treatment. In fact, libconvenience.a would have no effect on the final 'ar' command line for libtarget; it would/may only influence order-of-compilation due to dependency calculations.

(3) Note that because --exclude-libs is supported only by the GNU linker, and only for i386pe- and ELF- targetted ld's, on other platforms it should degrade to "pretend argument is just a non-installed static library". That is, -exclude-lib would

(a) ALWAYS "turn off" explode-and-incorporate behavior when building static libtarget. (This means that libconvenience.a is more-or-less removed from libtarget_LIBADD, as far as building libtarget.a is concerned, on all platforms)

(b) When building shared libtarget on a platform that supports symbol hiding (i386pe/ELF), do the --exclude-libs thing. Obviously, libconvenience.a is still be included on the link command for shared libtarget, sandwiched between --whole-archive / --no-whole-archive.

(c) when building shared libtarget where symbol hiding is NOT available, just "turn off" the --whole-archive / --no-whole-archive behavior. TBD: (c1) keep libconvenience.a on the link command line, and treat it like a "regular" static library.
    (c2) drop libconvenience.a from the link command line entirely.
The choice betwen (c1) and (c2) might be conditionalized on whether -no-undefined was specified, but that might be just a little too "cute".

Caveats: end user run into trouble is she explicitly decorates some symbols in libconvenience.a with declspec(dllexport) [i386pe] or perhaps sets visibility explicitly (ELF). Don't Do That. that I've gotten it all written down, it sounds very possible that this could give all the required features my "new library type" would, but without requiring any changes to the .la format, and putting the control in the hands of the target library (e.g. 'how is used') rather than in the hands of libonvenience itself (e.g. 'how is built'). That's a plus.

There's still some issues with convenience libs themselves (where the following discussion ignores ALL of the above, no new library types, no new -exclude-lib option), like:
 (a) we should build a pic version and stuff it inside .libs/ if
     -enable-shared, and put a regular version in . if -enable-static
     (with caveats for Linux/MIPS, etc, where everything truly is
     pic -- unlike i386pe where we just pretend that everything is pic
     and hope the Windows Runtime Loader fixes everything properly,
     with all the attendant .rdata problems)
 (b) When linking a shared lib where a convenience lib appears in the
     deps, use the one in the .libs/ directory
 (c) ??? when linking a PIE executable where a convenience lib appears
     in the deps, use the one in the .libs/ directory
 (d) when creating a static lib where a convenience lib appears in the
     deps, use the one in the . directory when doing the explode-and-
     incorporate thing.
 (e) when creating a next-level convenience lib that has THIS
convenienece lib as a dep, again, create a pic version in .libs and
     a non-pic version in .  In each case, use the "matching" version of
     the dependency when doing the explode-and-incorporate thing.
 (f) In all other cases, use the "regular" lib

[replace 'explode-and-incorporate thing' with 'partial linking thing' if your platform supports it]

These improvements to "regular" convenience libraries are, IMO, necessary for proper operation where pic/no-pic makes a difference. Plus, they'd make the -exclude-lib stuff work a lot better, too. <g>

But I'd rather not -- especially this close (?) to 2.0final

Certainly not.

Yeah, regardless of where this discussion goes, it's not likely to have any effect until after libtool-2.0-gold, and gcc-4.4+. So in the interim, gcc/win32 will have to do _something_ else. I'll keep hacking on it.

However, gcc steering committee policy is that external tools will be either exact release versions, or top-of-development-tree for the external tool. So, they would actually be *required* to use the development versions after libtool-2.0 comes maybe gcc can benefit from something like the above, sooner than libtool-2.2.


reply via email to

[Prev in Thread] Current Thread [Next in Thread]