Re: [RFC] New library "type" needed?

[Charles Wilson]

Ralf Wildenhues wrote:
> Here's a review based on the question how such a concept could be
> expressed portably.  I don't see how something like this would be of
> great help if it could not be made to gracefully decay into something
> still usable, on at least a useful set of platforms if not all of them.

Agreed. While I was considering the problem, I kept getting the feeling 
that either (1) I was missing something, or (2) this was very win32 
specific: it's a problem only for platforms that require -no-undefined 
for shared libraries, which I think is only win32 and AIX.

> * Charles Wilson wrote on Sun, Mar 25, 2007 at 09:34:47PM CEST:
> > What I really want is a type of convenience library (call it a "resolver" 
> > library?) that
> >
> > (1) is built both PIC and non-PIC, depending of course on the value of 
> > enable_shared and enable_static.  So far, just like a "normal" convenience 
> > lib.
>
> Well, currently a convenience lib is not built in both ways at the same
> time, but that's only a side issue for the discussion at hand.

Really? I thought I had both; perhaps it was just accumulated clutter 
from multiple builds with different settings.  In any case, I can see 
where the current behavior you describe could be problematic on some 
platforms, including win32, in certain cases.

> > (2) where the non-PIC resolver library is ignored when building dependent 
> > static libraries (and dependent static [dlpreopened] modules).  That is, NOT 
> > exploded and included within the dependent.  However, the non-PIC resolver 
> > library should be used like a normal convenience library when building 
> > dependent, in-package executables that depend on it directly
>
> OK so far.  You want a convenience archive without the whole-ness.
> Should it be possible to install this thing or not? 

Ummm...maybe?  Let me ping back a few more questions: how would you 
install/distinguish between the two versions (no-pic and pic)?  Would we 
need a naming convention of some sort and teach ld/gcc about it (a la' 
.dll.a)

This could get ugly.

On the other hand, currently binutils/gcc/etc install libiberty.a -- but 
do not install any headers for it.  So the only was to use an 
*installed* libiberty is to declare the relevant functions yourself (or 
go dig around in the src for the headers), AND hope that whoever built 
the libiberty.a you want to link against, built it in such a way that it 
includes all the functions you want.

In short, *currently* libiberty is not well-handled.  Since the intent 
for that library is to share at the source level, a better policy would 
be for binutils/gcc/etc to affirmatively change their build structure to 
 NOT install libiberty.a.

And nobody talks about installing a libgnulib.a -- especially as 
gnulib-tool lets you change the library name...

So maybe the answer to the question SHOULD BE no, even if at present the 
 actual practice IS to install libiberty.a.

> Can it itself have
> nontrivial (i.e., non-convenience) dependencies?

Errr...the two extant examples (libiberty and gnulib) do not, but...

> The first of these questions is really a crux of the matter: if libbfd
> can be shared and installable, then really I don't see any way except
> - either explicitly forbid any third party to use your library, even
>   though you install it shared,
> - or have at least some sort of stable interface notion.

Yes. Since libiberty is explicitly unstable in its interface, I believe 
it really makes no sense to install it; it's just of no use outside the 
build tree.

Following this line of reasoning, I think my "resolver" library concept 
is really just a variant of a convenience lib -- in the sense that 
convenience libs are not usually (ever?) considered as installable objects.

> > (worry about 
> > PIE here? We don't worry about it at present anywhere that I can see).
>
> We have very minimal PIE support only currently in CVS HEAD, in compile
> mode.

Don't know enough about this issue; I'll let it pass...

> > (3) where the PIC resolver library is used *like a static library* when 
> > building dependent shared libraries -- that is, used to satisfy undefined 
> > symbols in the shared library if -no-undefined, but where the objects in the 
> > PIC resolver library are not included wholesale via 
>
> [ "not" correction included ]
>
> > --whole-archive/--no-whole-archive -- and better yet, on win32 they should 
> > be excluded from auto-export using -Wl,--exclude-lib -Wl,<PIC resolver 
> > library name>.
>
> Lots of systems don't allow symbol hiding (or, at least, Libtool
> currently doesn't support it well on most systems).  What should happen
> for them?  More to the point: what if libbfd uses the resolver libiberty
> release x, while libfoo used release y of libiberty, we're on a system
> without symbol hiding, and my program links against both libbfd and
> libfoo?  The order of linkage will determine in which library the blow
> up occurs, be that silent or spectacular, depending on how incompatible
> x and y are.

But on how many systems are BOTH of the following true:
  (1) to build shared requires -no-undefined
  (2) but symbol hiding is unavailable

You know, thinking about this more, I guess (1b) from my previous email 
is not THAT bad -- for shared libraries; just a little bloat of the 
final library -- it requires a little platform-specific gobbledy-gook in 
configure.ac/Makefile.am, but libbfd (and libobject, etc) already do that:

bfd/configure.in from gcc trunk:
-------------------------------------------------------------
# Horrible hacks to build DLLs on Windows.
WIN32LDFLAGS=
WIN32LIBADD=
case "${host}" in
*-*-cygwin*)
  if test "$enable_shared" = "yes"; then
    WIN32LDFLAGS="-no-undefined"
    WIN32LIBADD="-L`pwd`/../libiberty -liberty -L`pwd`/../intl -lintl 
-lcygwin -lkernel32"
  fi
  ;;
*-*-linux*)
  # We borrow WIN32LIBADD so that the shared libbfd won't depend on
  # libiberty.a.
  case "${host}" in
  mips*-*-linux*)
    # Linux/MIPS uses PIC by default.
    if test "$enable_shared" = "yes"; then
      WIN32LIBADD="-L../libiberty -liberty"
    fi
    ;;
  *)
changequote(,)dnl
    x=`sed -n -e 's/^[  ]*PICFLAG[      ]*=[    ]*//p' < 
../libiberty/Makefile | sed -n '$p'`
changequote([,])dnl
    if test -n "$x"; then
      WIN32LIBADD="-L../libiberty/pic -liberty"
    fi
  ;;
  esac
  ;;
esac
AC_SUBST(WIN32LDFLAGS)
AC_SUBST(WIN32LIBADD)
-------------------------------------------------------------

Now, there are things in the snippet above I'd do differently (like: no 
need for cygwin and kernel32; shouldn't configure.ac just declare 
conditionals, and the actual additions to _LIBADD/_LDFLAGS be done in 
Makefile.am under the control of those conditionals; etc).

What's amusing though, is that what STARTED as a "horrible hack for 
win32" got hijacked not just by *-*-linux, but also by EVERY other 
platform in the default clause!

Anyway, this same machinery -- since it is there already -- can be used 
to insert -Wl,--exclude-lib in the right place, for win32.

I'd still like to be able to build my convenience library as both pic 
and non-pic tho.  And I still want to prevent libiberty.a(non-pic) from 
getting the --whole-archive treatment when it comes to libbfd.a.

Hmmm.  More on that, below.

> > Also, these "static" libraries should not trigger the "you 
> > can't create shared libs with static dependencies" filter within libtool.  
>
> Several systems simply don't allow to mix PIC and non-PIC symbols.
> On w32 this warning is pathetic, but on others we would have a problem
> here.  (I can see why, on w32, you'd want to kill the warning here.)

No, you're missing the point of my statement: I *know* you don't want to 
mix pic and non-pic. In fact, I'm sorta kinka trying to make win32 
behave that way, too, at least in this case, as an 'inoculate against 
the future' effort. (in case "true" -fpic support for win32 is merged 
into gcc trunk).

What I meant is, since in my hypothetical, we are being careful to have 
one .a full of pic symbols, and another .a full of non-pic symbols, then 
when we are creating a shared library and are explicitly using the .a 
full of pic symbols -- in THAT limited case, we should not get the 
FAILURE message about "you can't create shared libs with static 
dependencies"

This is not a warning, not even on win32.  It's a failure; libtool dies, 
make reports error, bang you are dead. This is because we currently 
can't tell, just from looking at the .a, if the objects are pic or 
non-pic.  It took a lot of dancing just to teach func_win32_libid() how 
to distinquish between import libraries and "regular" static libraries. 
Now, *if* there truly were a difference (on win32) between "pic" and 
"non-pic" objects, we'd have a fighting chance to educate 
func_win32_libid -- but at present, it's a distinction without a difference.

So, that's why I suspected that an implementation of my idea would 
require an additional keyword in the .la file -- so that libtool 
linkmode could tell (from the .la) that

"yes yes, I know file_magic_cmd reports that this is a "static library" 
and ordinarily that would be a problem, but since this particular 
dependency claims to be a "resolver", I know I can use the "pic" version 
(stored as the first element of library_names, where the .dll.a would 
usually go) without trouble.  So no warn, no fail..."

> > (Hmm. When -no-undefined is NOT specified, then there's a choice: you could 
> > satisfy as many undefined symbols as you could by passing the resolver lib 
> > to the linker, or you could simply drop the resolver entirely -- which would 
> > match the current build procedure for libbfd & friends on non-win32...)
>
> > I don't think it would be that difficult to add this facility to libtool -- 
>
> No, it wouldn't.  The semantics are the hard part.  If we can't get
> well-defined semantics, the new baby will be useful for binutils (and of
> course the set of packages that use it) only.

Possibly.  Also might be useful for packages that use gnulib.

But, it still smells to me like this whole problem might be a win32-only 
thing. Like THAT's new.  So, since win32 has symbol hiding, maybe a 
little bloat is not that bad: build as a regular convenience archive, 
libtool will include it entirely into the eventual target lib via 
--whole-archive, BUT a little configury/automake magic in these limited 
cases to finagle --exclude-lib into _LDFLAGS.

(again: okay for shared lib. But static target lib...)

So, instead of new library "type", is there some sort of directive we 
could implement instead?  Say, something like:

--------- Makefile.am --------
noinst_LTLIBRARIES = libconvenience.la
LT_LIBRARIES = libtarget.la

libtarget_LDFLAGS = -no-undefined -exclude-lib libconvenience.la
libtarget_LIBADD = libconvenience.la
-------------------------------

Where the semantics of the libtool option '-exclude-lib' would be

(1) arguments must be convenience libraries (if functionality is useful, 
later modifications could relax this)

(2) libconvenience.a is given the --whole-archive/-no-whole-archive 
treatment when building the shared libtarget, but ONLY if the target 
architecture supports symbol hiding (see below).  libconvenience.a is 
also marked with "-Wl,--exclude-libs -Wl,libtarget.a".

(2a) libconvenience.a is included in dependency calculations and 
everything for the static libtarget.a, but is NOT given "explode and 
incorporate into libtarget.a" treatment.  In fact, libconvenience.a 
would have no effect on the final 'ar' command line for libtarget; it 
would/may only influence order-of-compilation due to dependency 
calculations.

(3) Note that because --exclude-libs is supported only by the GNU 
linker, and only for i386pe- and ELF- targetted ld's, on other platforms 
it should degrade to "pretend argument is just a non-installed static 
library".  That is, -exclude-lib would

  (a) ALWAYS "turn off" explode-and-incorporate behavior when building 
static libtarget.  (This means that libconvenience.a is more-or-less 
removed from libtarget_LIBADD, as far as building libtarget.a is 
concerned, on all platforms)

  (b) When building shared libtarget on a platform that supports symbol 
hiding (i386pe/ELF), do the --exclude-libs thing.  Obviously, 
libconvenience.a is still be included on the link command for shared 
libtarget, sandwiched between --whole-archive / --no-whole-archive.

  (c) when building shared libtarget where symbol hiding is NOT 
available, just "turn off" the --whole-archive / --no-whole-archive 
behavior.  TBD:
    (c1) keep libconvenience.a on the link command line, and treat it 
like a "regular" static library.
    (c2) drop libconvenience.a from the link command line entirely.
The choice betwen (c1) and (c2) might be conditionalized on whether 
-no-undefined was specified, but that might be just a little too "cute".

Caveats: end user run into trouble is she explicitly decorates some 
symbols in libconvenience.a with declspec(dllexport) [i386pe] or perhaps 
sets visibility explicitly (ELF). Don't Do That.

Hmmm...now that I've gotten it all written down, it sounds very possible 
that this could give all the required features my "new library type" 
would, but without requiring any changes to the .la format, and putting 
the control in the hands of the target library (e.g. 'how 
libconvenience.la is used') rather than in the hands of libonvenience 
itself (e.g. 'how libconvenience.la is built').  That's a plus.



There's still some issues with convenience libs themselves (where the 
following discussion ignores ALL of the above, no new library types, no 
new -exclude-lib option), like:
 (a) we should build a pic version and stuff it inside .libs/ if
     -enable-shared, and put a regular version in . if -enable-static
     (with caveats for Linux/MIPS, etc, where everything truly is
     pic -- unlike i386pe where we just pretend that everything is pic
     and hope the Windows Runtime Loader fixes everything properly,
     with all the attendant .rdata problems)
 (b) When linking a shared lib where a convenience lib appears in the
     deps, use the one in the .libs/ directory
 (c) ??? when linking a PIE executable where a convenience lib appears
     in the deps, use the one in the .libs/ directory
 (d) when creating a static lib where a convenience lib appears in the
     deps, use the one in the . directory when doing the explode-and-
     incorporate thing.
 (e) when creating a next-level convenience lib that has THIS
     convenienece lib as a dep, again, create a pic version in .libs 
and
     a non-pic version in .  In each case, use the "matching" version of
     the dependency when doing the explode-and-incorporate thing.
 (f) In all other cases, use the "regular" lib

[replace 'explode-and-incorporate thing' with 'partial linking thing' if 
your platform supports it]

These improvements to "regular" convenience libraries are, IMO, 
necessary for proper operation where pic/no-pic makes a difference. 
Plus, they'd make the -exclude-lib stuff work a lot better, too. <g>

> > But I'd rather not -- especially this close (?) to 2.0final
>
> Certainly not.

Yeah, regardless of where this discussion goes, it's not likely to have 
any effect until after libtool-2.0-gold, and gcc-4.4+.  So in the 
interim, gcc/win32 will have to do _something_ else. I'll keep hacking 
on it.

However, gcc steering committee policy is that external tools will be 
either exact release versions, or top-of-development-tree for the 
external tool. So, they would actually be *required* to use the 
development versions after libtool-2.0 comes out...so maybe gcc can 
benefit from something like the above, sooner than libtool-2.2.

--
Chuck