Re: libltdl exports no symbols (cygwin)

[Charles Wilson]

Eric Blake-1 wrote:
> > > > I am a bit wary that it may cause regressions in users' packages, as
> > > > they may be using LT_SCOPE.  But I see that with the other change, this
> > > > is needed.
> > > User-package use of internal libtool macros (e.g. LT_SCOPE) is outside 
> > > libtool's control.
> > Hmm, yes.  I've never been certain whether LT_SCOPE was *intended* to be
> > purely internal, or for the user to use.  The latter seems inappropriate
> > to me now, but I don't think people were aware of this all the time.
>
> Indeed - ever since this patch, CVS M4 is now broken on cygwin.

Err...and this is news? <snark>For over three years, CVS M4 has had 
serious issues on cygwin.  Every six months or so, I used to give it a 
go, but finally gave up -- as I have said before, making M4 depend upon 
the dustiest corners of libtool/libltdl functionality was a serious 
misdesign for such a fundamental tool.  You can't have a working m4 2.0 
unless libtool/libltdl on your platform fully supports modules, in both 
compiled-in (dlpreopen) and externally loaded form (dlopen/LoadLibrary). 
 But without m4 2.0, you won't have working autoconf-NEXT-MAJOR-VERSION 
-- which may or may not be required for the libtool version that 
actually supports modules on your target platform as described above. 
Words fail me...</snark>

>  I am
> getting
> all sorts of link errors along the lines of:
>
> libtool: link: gcc -shared  modules/.libs/gnu.o   m4/.libs/libm4.dll.a    -o
> modules/.libs/gnu-0.dll -Wl,--enable-auto-image-base -Xlinker --out-implib
> -Xlinker modules/.libs/gnu.dll.a
> modules/.libs/gnu.o: In function `m4_regexp_compile':
> /home/eblake/m4-head/modules/gnu.c:140: undefined reference to
> `_rpl_re_set_syntax'
>
> This is because M4 was relying on auto-imports, and now that libltdl
> has an explicit export, the auto-import mechanism is disabled.

Let's be very precise with terminology here.  M4 may or may not be 
relying on auto-imports -- but that is beside the issue.  The problem is 
that auto-EXPORT when linking (e.g. creating) the DLL (in this case, 
m4.dll) is turned off when any .o or .a(e.g. convenience archive) 
included in the link contains symbols which are explicitly exported.

In this case, when building the libm4 module, DLL_EXPORT is defined -- 
which triggers the LT_SCOPE == declspec(dllexport).  This is bad, since 
we expect that the libltdl convenience archive contains _foo, not 
__imp__foo.  [[ However, it appears that the "convenience" archive 
actually contains declspec(dllexport)-decorated symbols...this is 
because convenience archives are compiled with the same options as the 
eventual target: in this case, the m4 dll. It is "pic", so the libltdl 
convenience archive is "pic" -- which in this context means 
declspec(dllexport). ]]

The simplest workaround for CVS M4 is to '#define LT_SCOPE extern' 
before including ltdl.h when building libm4 DLL, AND to do the same when 
building the libltdl library *as a convenience archive*.  This will 
ensure that the symbols in libltdl are not decorated, and that none of 
the components linked together to create libm4 DLL contain dllexport-ed 
symbols; thus re-enabling auto-EXPORT behavior when linking the m4 dll.

> I still think the fix to libtool was the correct thing to do, but in order
> to fix M4, I need some help.  It looks like I need to properly mark for
> export all symbols intended for clients of libm4.dll to use, since libm4.dll
> utilizes libltdl.

Well, maybe...or not -- I think the workaround above is easier than all 
THAT.

> > Eventually, we should do something about this by offering a portable way
> > to decorate imports/exports for users of libtool.  Not right now.
>
> That's a shame.  Until libtool provides helper macros that automatically
> do the right thing when compiling with libtool, making M4 work on
> cygwin will be a tougher process.

But libtool already does: you can use a combination of DLL_EXPORT and 
BUILDING_MYLIBARY to Do The Right Thing.

The real issue is that the "default" behavior that libtool expects, on 
cygwin/mingw -- no special -Ddefines -- is static libraries.  However, 
on modern systems, users and developers alike universally expect shared 
libraries (regardless of the fact that on ELF systems, you have to pass 
special flags to your compiler to CREATE those shared objects: -fPIC, 
etc).  The fact is, if I'm building foo which requires libbar, I 
*usually* expect that libbar is a shared library.  On ELF systems, this 
is no big deal: I #include the headers, and link to -lbar, and it Just 
Works(tm) -- I'd be surprised if my foo executable didn't depend on 
libbar.so!

The same *expectations* _should_ apply on cygwin/mingw: if foo uses 
libbar, it should #include libbar's headers and link -lbar, and get a 
foo.exe that depends on bar.dll.  That should be our default assumption. 
 If I want to link against the static libbar, I should have to 
explicitly do that.  The linker DOES work this way: it prefers 
libbar.dll.a (and even bar.dll) over libbar.a -- unless you explicitly 
say -Bstatic.  IMO, the compile process should be the same way: no 
special -Ddefines, you get the DLL import symbols.

But that isn't the way things work, for historical and practical 
reasons.  And short of something really clever:
  (1) auto-import
  (2) Bruno's ideas
making DLLs "Just Work" the way I have described requires significant 
effort on the part of the library porter.  Which is why everybody jumped 
onto the (1) bandwagon once it was implemented.  If (2) had happened 
first, everybody would've jumped to it.  However, as things stand (2) 
has to compete against entrenched inertia of (1)...

Back in the dawn of time (the glorious days of cygwin B20.1, which as 
everyone knows was practically perfect in every way) I ported a lot of 
libraries to cygwin, and "dll-ized" them.  Back then, I used a magic 
stanza like this (there might have been an 'extern' here or there):

#ifdef __CYGWIN__
# ifdef LIBZ_STATIC
#  define LIBZ_IMPEXP
# else
#  ifdef BUILDING_LIBZ
#   define LIBZ_IMPEXP declspec(dllexport)
#  else
#   define LIBZ_IMPEXP declspec(dllimport)
#  endif
# endif
#else
# define LIBZ_IMPEXP
#endif

::: libtool was not an issue, because it really only barely
::: supported DLLs at the time.  That came later, with the
::: advent of auto-import and pseudo-relocations (more below).

And then I manually decorated all the exportable symbols with 
LIBZ_IMPEXP (with, of course, accompanying changes in the 
Makefiles/configury).  Obviously, these changes were massively intrusive 
-- BUT, in this scheme, once the library was built, clients by default 
would get the dllimport symbols, and since the linker would by default 
use the import lib, all was well.  Or, a client could
  -DLIBZ_STATIC
when compiling and
  Wl,-Bstatic -lz -Wl,-Bdynamic
when linking, to get a static link.  (Yes, it was tricky and ugly -- 
BUT, in this scenario, linking statically was the UNUSUAL operation. 
The TYPICAL dynamic link was clean).

(I actually got tired of -DTIFF_STATIC -DJPEG_STATIC -DZLIB_STATIC ... 
and added an extra symbol to all of the ports: -DALL_STATIC. But still...)

And then along came auto-import...and most of these porting efforts 
atrophied away, as they were no longer necessary in most cases (assuming 
you ignore Bruno's critique of auto-import's memory inefficiencies).

Then, along came pseudo-relocation support in cygwin and mingw runtimes 
-- which supported auto-import of multiword variables, the only missing 
piece above.  That made the rest of the declspec()-for-cygwin ports 
evaporate (ignoring the problems introduced by gcc-3.3's .ro section 
semantics whcih don't mix well with psuedo-relocation.  Strangely, it 
was Bruno's objection to my "fix" for this problem in gettext which lead 
to his really bright idea (2) above.)

And then, with these two features, libtool 1.5.x could actually be used 
to build DLLs on win32/cygwin fairly easily, without a lot of intrusive 
patches to the library code (which was SO intrusive that many upstream 
maintainers would not even consider accepting such patches).

The only folks still messing around with declspec() in the 
autotool/libtool world were (a) diehards (b) folks using mingw 
explicitly to build MSVC-compatible DLLs -- obviously, this only works 
for non-mangled ABIs: e.g. C only (c) people using GNUish shells to 
drive msvc/cl/lib/link tools (d) hybrids: DLLization performed first for 
msvc, and then re-used by the mingw build machinery added later.  Very 
fragmented and unsystematic.  No "standard" mechanism ever developed.

The closest to a "standard" was the mechanism described in the original 
version of the AutoBook in the section concerning w32 ports.  This 
solution used libtool-1.4x and explicitly relied on the 1.4x-era 
-DDLL_EXPORT!  (-DDLL_EXPORT was removed for cygwin after then 
introduction of auto-import, until re-introduced last June. It has 
always remained for mingw.)

> > > > I'm still wondering what to put in NEWS for this and the depending
> > > > change.  We should mention this somehow.  Suggestions appreciated.
> > > "Fix regression in libltdl symbol exports on Cygwin. Side effect: 
> > > LT_GLOBAL_DATA and LT_SCOPE are now explicitly defined as 
> > > declspec(dllexport), bypassing auto-export logic on Cygwin.  This tracks 
> > > existing behavior on MinGW."
>
> Should we also mention the side effect that you must now mark
> explicit exports, since you can no longer rely on 

> auto-imports?

auto-EXPORTs, actually.  The problem only occurs when:
  * creating a DLL that itself links in
  ***  a libltdl.a convenience archive
But the problem is only *detected* when linking to this poor disabled DLL.
[***]

auto-import still works -- it's just that there's nothing exported by 
the DLL for auto-import to use.  And, there's a fairly simple 
workaround, that can be imposed manually as described above.  It could 
even be automated by libltdl: if libltdl is being built as a convenience 
archive, define some symbol.  In ltdl.h (lt_system.h in HEAD), use that 
symbol to force LT_SCOPE to be 'extern' even on CYGWIN|WIN32, thus 
overriding the DLL_EXPORT from libtool.

--
Chuck


[***] Note that this problem should be REALLY rare: it only happens when
  (1) you have a set of library code (call it 'A') that includes 
necessary support for building as a dll
  (2) BUT you're actually using it as a convenience library when 
building a DIFFERENT dll (call it 'B')
  (3) AND "A" and "B" each follow different standards for symbol 
decoration (e.g. libm4 dll has no symbol decoration whatsover; but ltdl 
follows a -DDLL_EXPORT-driven scheme)

You do NOT run into this problem [you get a different issue!] when 'A' 
is built as an external, static library.  In that case, libtool will 
refuse to create 'B' with "static" dependency "A", thanks to its 
justified (on ELF) paranoia about mixing "PIC" and "non-PIC" code.  Yes, 
yes, there's no real difference on win32 -- but cygwin-libtool still 
honors this paranoia (as does mingw-libtool, I think).