Re: [RFT PATCH v4 0/8] Sysroot series

[Charles Wilson]

On 8/7/2010 2:36 AM, Ralf Wildenhues wrote:
> FWIW, I didn't find any more objectionable things in the rest of this
> patch series, so whenever you have something that addresses the known
> issues, I think a public branch would be fine. 

Well, that's up to Paolo.

> If merely additional
> testsuite items are missing, that shouldn't hold up creating a branch,
> only merging it.

Hmm...but if I understand your last message in the other subthread,
adding a test for sysroot with "native" compilers is very very optional,
given the fact that we have not yet found ANY default system toolchains
that actually support --sysroot.  If /that/ is the 'testsuite item [that
is] missing', it really shouldn't hold anything up, should it?

Similarly, the long subthread concerning MinGW + DESTDIR...is not really
germane to this patch at all, is it?  So, there's really nothing to
"test" that ought to be in THIS patch series, nor should those
consideration hold up merging this patch series' branch into master.


Reviewing the thread, here's a preview of items "still outstanding",
detailed in the long discussion below:

1) done
2) done
3) issue with '# Collect and forward deplibs of preopened libtool libs'
   In progress? Anything?
4,5)
   * patch for old libtool, to not barf on '='
   * TBD mechanism to strip '=' from .la after deployment on $host
6) support in libltdl for '=' in .la files


=====================
1) ln -fs non-portable

Ralf said:
> Also, in 'ln -sf' in the testsuite addition, the -f is non-portable
> to Solaris, and for DJGPP (cough) it would even need $(LN_S) but let's
> ignore that for now.

Paolo said:
I'm squashing this [sysroot.at]:
-  ln -sf "$gcc_sysroot/$file" "$sysroot/$file"
+  rm -f "$sysroot/$file"
+  ln -s "$gcc_sysroot/$file" "$sysroot/$file" || \
+    cp "$gcc_sysroot/$file" "$sysroot/$file"

=====================
2) Warnings when compiling sysroot.at files

Ralf said:
| ++ make all
| stderr:
| lib2.c: In function 'g':
| lib2.c:6:13: warning: initialization makes pointer from integer
without a cast
| stdout:

I said:
> This addendum appears to do it (Tested on cygwin->mingw cross.
AT_DATA([lib2/lib2.c]:
+extern char *f(const char *s);
AT_DATA([prog/prog.c]:
+extern char *g(const char *s);

Paolo said:
> Wquashed, thanks.

=====================
3) Ralf:
> the part of ltmain starting with
>       # Collect and forward deplibs of preopened libtool libs
> will source .la files but looks like it mistreats = in
> $dependency_libs

Paolo:
> OK.

but...no further communication on this point. Paolo, are you working this?

=====================
4) Cleaning up '=' when deploying .la files to their $host

Ralf:
> - is it intended that the = remain in the .la files after these are
> moved to their final location on the host system?  Or should
> --mode=finish remove them?

Paolo said (A):
> I thought so, but --mode=finish takes a directory. :/
> Should I do that on all .la files?

Ralf replied (to A):
> Isn't that what you actually need though?
>
> Let's turn the question around: if some of the installed .la files
> have been fixed already, won't the packages installed afterwards
> then expose libtool bugs again, because they don't get handled by
> your patch series?

I replied (to A): (B)
> Wait, I thought you NEEDED the '=' marker in there, or the .la files
> wouldn't give correct behavior when they were used from within a
> sysroot.
>
> Sure, you might want to strip out the '=' flags if you
> copied/installed the built package over onto $host and wanted to do
> "native" development over there, (snip)

Ralf replied (to B):
> Yes, but isn't --finish supposed to be run only once these files are
> not within a sysroot any more?
...
> Exactly. For some use cases it is "normal" to have this native
> development later.  (Example: most distribution packagers.)

And that's where the subthread ended.  However, I have a new data point:
GNU ld supports '=' in arguments to the -L option -- even if your GNU
linker was compiled with --sysroot support (in which case, the '=' is
just ignored).

Unfortunately, I couldn't find any evidence that (non-sysroot) GNU ld
extends similar treatment to -R or -rpath arguments, and it would
certainly be confused by this:
   dependency_libs=' =/mingw/lib/libfoo.la'


<Devil's advocate:>
Given all of that, it seems to me that the "correct" solution here is

  1) libtool --mode=finish should remove the '=' from .la files

  2) BUT this should be done as part of creating the installable
     package for direct deployment on $host, NOT when installing into
     $build's sysroot or creating an installable package for someone
     ELSE to install in a similar $build's sysroot (e.g. a mingw-libfoo
     RPM that is part of a mingw-focused SDK for use on Fedora)

It would be nice if #2 could be done simply by telling system
integrators to add:

    %post
    /usr/bin/libtool --mode=finish %libdir

to the package's rpm.spec for instance.  As another example, suppose I
use a $build=linux, $host=cygwin compiler with a sysroot.  I build
package "libfoo".  I'd package it two ways:

   1) I'd create a cygwin-libfoo RPM exactly as it should appear
      in my /usr/i686-pc-cygwin/sys-root/ area. I'd make no changes
      in any of the files.

   2) I'll create a libfoo .tar.bz2 for cygwin's setup.exe to
      install on "native" cygwin.  Again, I wouldn't change the files
      (so they'd still contain the '=' stuff) But, this package would
      also include an /etc/postinstall/libfoo.sh script (cygwin's
      setup.exe's mechanism for handling postinstallation actions).
      This script would have:
         #/bin/sh
         /usr/bin/libtool --mode=finish /usr/lib

The problem with THAT is, right now finish takes a directory. Many of
the finish_cmds for various platforms act specifically on dirs, so
that's ok (e.g. finish_cmds='PATH="\$PATH:/sbin" ldconfig $libdir').
Others have a finish_eval that iterates through the files found on the
deployed system:
        finish_eval='for lib in `ls $libdir/*.ixlibrary.....'

Now, suppose we added, for ALL platforms, an additional element to
finish_[cmds|eval], like this:
  for f in `ls $libdir/*.la`; do fixup_sysroot_markers "$f" ; done
(where $libdir is replaced by the arguments specified when --mode=finish
is invoked, obviously)

Well...that's not really appropriate.  We install a package libfoo that
puts a single '='-polluted .la into /usr/lib -- and then we "fixup"
EVERY .la file in that directory?

I think --mode=finish needs to accept additional arguments that are not
directories, somehow.  Maybe mode=finish needs to iterate through its
arguments, and split them into "directories" and ".la files", and then
execute two different commands:

The existing finish_[cmds|eval] on the list of directories, and

Some new function/cmd/eval on the list-of-la-files. Maybe rather than
indirected via an m4-defined variable, always just explicitly invoke a
specific fixup_sysrooted_libtool_libs() function.
</Devil's advocate>

That's just ugly.  A whole lot of work...is it really necessary?  What
if simple fixes were available to all/most widely-used versions of
libtool, so that they could just "deal with" '=' in the .la files?

That's fine, for libtool the script.  But what about executables that
use libltdl?  Do we expect everybody to upgrade their libltdl shared
libraries?

What if somebody linked an old one with a static libltdl? (Well, you
could argue that they shouldn't be trying to load a brand new library
built using the very latest bleeding edge libtool, with an old libltdl
system, but...that's not really fair.)

And, in these corner cases, they could always manually remove the
offending '=' marks.


=====================
5) Retrofit support:

Ralf asked:
> should we provide a minimal patch for vendors 2.2.10 (or earlier
> versions, maybe even 1.5.x) that makes libtool not barf upon = in .la
> files, for smoother upgrades (given that 2.2.12 could contain other
> issues not making for a smooth upgrade path)?

Paolo said:
> Yes, especially if --mode=finish does nothing.  I'll see who maintains
> libtool at Red Hat, maybe he can relieve me of that part. :)

Now, I said:
> I'm starting to think it really shouldn't strip out the =. So a
> compatibility patch for older libtool is probably a good idea.
>
> BTW, does libltdl need to know about these '=' markers, when it tries
> to load a module?

Unfortunately, the second paragraph of my statement appears to negate
the first.  Since we can't expect everybody to run out and recompile
libltdl and/or libltdl clients, simply because we *changed the default
.la file format* -- we need to provide a way for the .la files to be
normalized so that, on the $host, existing libltdl clients continue to
work.  (Regardless of whether we provide a small patch to "libtool" so
that native development on $host isn't confused by any '=' retained in
.la files).

Ralf appears to agree. He said:
> Good point, yes [libltdl needs to know about '=']

and in response to my statement that 'it seems to me that [cleaning up
'=' from .la files] is a packaging/integration task (e.g. not libtool's
problem), Ralf said:

> That's exactly what I'm trying to find out.  If it's a packager task,
> it needs to be documented as such, and if --finish is not the right
> place to do it, then we should invent another tool to easily to it,
> because we can't expect people to get those sed scripts right
> themselves.

So, taking (4) and (5) together, it looks like we need

a) backport support to older libtool so that, IF there are '=' symbols
in .la files, libtool won't barf when you're doing native development on
$host.

b) some tool (an extension to mode=finish that accepts a list of .la
files; some other tool?) to remove those '=' markers -- so that "old"
libtool, and apps using "old" libltdl, are not confused.  This tool
would be used as part of postinstall scripts in packages, or perhaps
done in a staging area on $build when preparing the image for creating a
deployment package (RPM, tarball, etc) for $host.  Whatever this is, it
needs documenting in NEWS.

=====================
6) libltdl support for '=' in the .la files, and/or other sysroot semantics.

Ralf says:
> Good point, yes

=====================
So, to sum up:

1) done
2) done
3) issue with '# Collect and forward deplibs of preopened libtool libs'
   In progress? Anything?
4,5)
   * patch for old libtool, to not barf on '='
   * TBD mechanism to strip '=' from .la after deployment on $host
6) support in libltdl for '=' in .la files

So, Paolo -- progress on any of these items? Do you need help
implementing any of them?

Ralf: do ALL of these items need solutions before the temp branch is
created?  If Paolo wants to delegate, it'd certainly be easier to manage
if the basic set of patches were already in a branch, which effort
continues on these items.

--
Chuck