Re: [RFT PATCH v4 0/8] Sysroot series

[Charles Wilson]

On 8/4/2010 2:15 PM, Ralf Wildenhues wrote:
> * Charles Wilson wrote on Tue, Aug 03, 2010 at 07:22:46AM CEST:
>> Now, since I've used --prefix=/mingw and NOT
>> --prefix=/usr/i686-pc-mingw32/sys-root/mingw, I can "cheat" even on
>> win32.  I can take the binary runtime components -- even the -devel
>> components! -- and install them in
>>    C: (or D: or E:)
>> and they'll end up under
>>    C:/mingw
>>    D:/mingw
>> or whathaveyou.  Thanks to a quirk in window's path resolution, any
>> unixish absolute path is interpreted as if it were a path from the drive
>> root of the CWD.  So, if CWD is on C:, then /mingw/libexec is
>> interpreted as C:/mingw/libexec.  Since ALL the unixish paths will start
>> with /mingw, this will Do The Right Thing.
> 
> Sorry, but I don't think that is viable.  Say, C:/mingw is owned by the
> administrator, but D:/mingw is domain of user A.  Starting your app on
> D: as user B makes B susceptible to files which A owns.  In general, you
> cannot leave away the drive letter when specifying the --prefix.

Then you are stuck with building twice.  Once for your sysroot
installation, and once for the "real" installation on $host.
However...that would STILL break:

libfoo.la
a)  compiled for sysroot using the cross compiler, and installed there,
    with --prefix=/something and --with-sysroot
b)  compiled for $host using the cross compiler, and installed THERE,
    with --prefix=E:/mingw or whatever.

Then, you compile libbar.la which depends on libfoo:
a)  compiled for sysroot using the cross compiler, and installed there,
    with --prefix=/something and --with-sysroot.  libbar.la(a)'s
    dependency_libs will look like
        -L=/something =/something/lib/libfoo.la
    This is obviously useful for future compilation of libbaz.la.

b)  compiled for $host using the cross compiler, and eventually
    installed there, with --prefix=E:/mingw or whatever.  However...
    libbar.la(b)'s dependency_libs will look like
        -L=/something =/something/lib/libfoo.la
    too!  But that's clearly wrong, if you want to copy libbar.la(b)
    directly over to $host and use it there, under E:/mingw.

Now, this is cross-compiler 101, for all architectures.  You really need
to use the same "prefix" underneath your sysroot that you use on the
actual $host.  Unfortunately, there is no such thing, on unix, as a
prefix named "E:/somedir" (note the missing initial '/').


But, that is untenable: on the one hand you want to assert that "thou
shalt include X: in thy prefixes when configuring for $host=mingw" even
when cross-compiling for any purpose, but then there's this impossible
requirement that the "prefix" underneath the sysroot be identical to the
"prefix" on $host -- otherwise this whole cross compile thing falls
apart, once you start chaining dependencies.

Now, if "/something" in the (a) cases above were "/mingw" (e.g.
"matched" the drive-letter-stripped version from the (b) cases above),
then it would actually work correctly -- but, you'd be open to exploits
from D:/mingw.


And this problem is not limited to libtool's .la files; if any path
containing $prefix is compiled into a binary, or recorded anywhere,
you'll have this problem.

The only real solution, as I've said, is:

  1) don't cross compile for $host=mingw. Ever. Always compile
     natively, and use --prefix=X:/stuff WITH the drive letter.

  2) Fix *every* package that records ANY directory information
     so that it is completely relocatable. Each binary or script, or
     tool that parses a text file containing directory information,
     should autodetect its installation location, and deduce relative
     paths to its resources from that.

#2 is a gigantic job, and it's not libtool's task to do that for the
universe of software packages. Now, MAYBE, we might take a look at
relocation support for .la files -- but I fear that would open us up to
more security holes than you could shake a stick at.  Resolving all
paths into relative paths from whatever is recorded in libdir=''? Oh, boy...


#1 is clearly unacceptable as well.

Hence...I think the consensus is:

  I) if you're compiling natively for mingw, use --prefix=X:/foo,
because it avoids the possible security issues.  And, it's just
objectively the correct thing to do.

  II) if you're cross-compiling -- whether for installation into
      the $build's sysroot, or for actual deployment onto $host -- use
      --prefix=/unixish/path, and make sure it matches in both cases.
      You'll exploit a quirk -- AND open your users up to a possible
      security exploit -- but there currently is no other viable way
      to do cross-compiles for $host=mingw with chained dependencies,
      such that they would actually WORK.

> Sure.  I don't disagree with leaving the sysroot part out of --prefix.
> 
>> However, items built for case (2) -- direct deployment on the $host (or
>> compiled "natively" ON the mingw host, that is MSYS/MinGW) -- are by
>> convention configured as:
>>    configure --prefix=`cd /mingw && pwd -W`
>> which ends up as
>>    configure --prefix=C:/MinGW
>> and the installation is usually preformed as
>>    make install prefix=/tmp/something/MinGW
> 
> I guess what I don't see is why the former case doesn't also use
>   --prefix=C:/MinGW

Because if you then try to reconstruct the actual location on the disk,
on $build under sysroot, you get:
   /usr/i686-pc-mingw32/sysrootC:/MinGW/
and now, the word "sysrootC:", besides being a sibling, rather than a
child, of the ACTUAL sysroot returned by the compiler via
-print-sysroot, is unsplitable: how can libtool extract the "C:/MinGW"
part when it is geared towards splitting paths at path-delimiter characters?

Because "C:/MinGW" doesn't start with "/", $sysroot$prefix is not a
reversible operation.  And the reversibility is a KEY requirement for
any sane sysroot handling -- by libtool or pkg-config or anybody else.

> [...]
>> Anyway, this workflow ensures that the actual, DOS-ish paths are
>> embedded anywhere such things are hardcoded, which is both good and bad.
>>  But it also ensures that there IS no concatenation of paths that might
>> start with X:, since we replace the ENTIRE prefix with a unixish one
>> during make install.  This technique works, even today on cross setups,
>> if you build one package at a time for deployment on $host.
> 
> Ahh.
> 
>> But you
>> can't easily do a "chain" of such builds on a cross system, without
>> effort.  You sorta have to do twice the builds: one of type (1) and one
>> of type (2), for each library in the chain:
>>
>>   a) zlib class (1) installed on $build in sysroot under $prefix
>>   b) zlib class (2) installed on $host under $prefix
>>   c) libpng class (1) installed on $build in sysroot under $prefix
>>      this will link against the libz.dll.a from (a).  If you have
>>      support on $build for running $host binaries (wine?) then at
>>      runtime it would use the libz-1.dll from (a) as well.
>>   d) libpng class (2) installed on $host under $prefix
>>      this will link against libz.dll.a from (a)!!!  However, once
>>      installed on $host, it will RUN against zlib-1.dll from (b).
>>   and so on, and so on.
>>
>> Using method (2) alone would not yet be appropriate for a case (1)
>> installation (e.g. into a cross $build system's sysroot), because then
>> you WOULD be concatenating a unixish sysroot with a dosish "prefix".
>> (e.g. you can't really use (b) and install it on $build in sysroot, and
>> pretend that's a suitable replacement for (a); ditto (d) vs. (c)).
> 
> Is this still relevant in light of my above reply?

Actually, I think it's even MORE relevant.  Short version:

1) cross-compiler-101: --prefix under $sysroot must == prefix on $host.
   Otherwise lots of stuff breaks.

2) for $host=mingw, the prefered prefix formula is a dos path, X:/foo.

3) but you can't use that formula under $sysroot on $build, because
   $sysroot$prefix where prefix=X:/foo is not a reversible operation

4) so we are stuck with a workaround for $host=mingw cross compiles,
   where we exploit a flaw in Windows path resolution: "unix-style"
   absolute paths (that start with /) are always interpreted with
   an implied X: drive spec, where 'X' is derived from CWD.  While
   this is a possible security hole, it is also the only current
   mechanism to actual build chains of dependent projects using a
   cross compiler for this $host.

>>> If yes, then
>>> very cool, and I'd very much like this to be exposed in the testsuite
>>> (only on w32 systems, of course, and the C:/... path should be something
>>> unlikely to clash with any actual user files, in case make install does
>>> write there after all).
>>
>> I'm not entirely sure what you want to test here -- or that it would be
>> something that *libtool's* testsuite should cover (wouldn't it be more
>> appropriate as part of automake's?)
> 
> Please allow me to give a sort-of simple-minded answer, hope I can make
> the gist of it clear:
> 
> If Libtool gains the --with-sysroot feature, then that should be tested
> as part of the Libtool testsuite.

OK.

>  Currently these tests all skip if the
> compiler found is not GCC or has not been configured with a sysroot.

Right.

> These two side-conditions mean that when I run the testsuite on all the
> systems and in all the configurations that I have been testing
> prereleases in the past, *none* of them will actually cover any of the
> new sysroot functionality, because even the cross compilers that I test
> do not use a sysroot.

Ah -- but gcc supports --sysroot even if it wasn't compiled with one (my
cygwin native 4.3.4 doesn't have a -print-sysroot, tho; so I wonder if
the --sysroot=/test/dir actually works for that compiler!)

However, my linux native compiler supports both (and, of course, wasn't
actually compiled with a sysroot of its own):

$ gcc --sysroot=/bob -print-sysroot
/bob

So, you could easily create a test that uses the native system compiler
to "fake" a cross-toolchain "sysroot" test. But to make that work you'd
still have to create a temporary fake sysroot underneath /bob as a
symlink farm using /usr/lib and /usr/include -- which would quickly
become painful, since most systems have LOTS of stuff under those two
directories...

And then there's the fact that the toolchain ITSELF has bits installed
under /usr/lib/gcc/*/...

> Given the experience that new Libtool tests need on the average about
> three or more iterations to get right portably, this makes me a bit
> nervous.  We should really, really try to get test coverage high for
> this feature on as many setups as possible.

Which definitely argues AGAINST making --with-sysroot the default, even
if you ARE using a cross compiler, right away.  The key thing is, we
want to ensure that

  1) native compilers don't break -- that's covered by the rest of the
     test suite

  2) cross-compiler setups don't break, when libtool is used in its
     default --without-sysroot (--with-sysroot=) mode.  I think this
     is covered by "the rest of the test suite" when you build and
     test libtool using a cross toolchain that has no sysroot.

Last -- and frankly least important in comparison to keeping EXISTING
functionality working -- is the

  3) test the new --with-sysroot support in conjunction with cross
     toolchains that actually have a sysroot.  This is covered by the
     three new tests -- but requires toolchains with sysroot support.

The only answer for (3) is "Get Ralf Access To More Cross Toolchains".

FYI, the Mandriva 2010 mingw32-gcc package is built with a sysroot;
there's a VMware image here:
  http://www.vmware.com/appliances/directory/va/434563
but you'd probably have to then do 'urpmi --fuzzy mingw' from inside it
as root to install the cross compiler.  If VirtualBox is more your
style, the newest pre-packaged image I found was Mandriva 2009, and I
don't know how its mingw cross compiler was configured.  So you'd
probably be better off converting the vmware image of 2010 to vdi format
using this procedure;
http://linuxpoison.blogspot.com/2010/06/how-to-convert-vmware-image-vmdk-to.html

> If there is anything in Automake that isn't working for this kind of
> test, well, then let's not use automake for this part of the Libtool
> testsuite.  We've been avoiding it in most of the new testsuite already
> anyway.

Well, my confusion was you seemed to be concerned that 'make install
DESTDIR=...' doesn't work with $host=mingw (or, at least, when
--prefix=D:/Dos/Abs/Path is used).  But DESTDIR is a creature of
automake.  libtool has some support for it, but the "breakage" is
fundamental to the design of the DESTDIR thing: all install paths are
immediately prepended by $(DESTDIR).

That's just broken, unless $prefix begins with a directory separator.
(Even if you cheat and ensure that DESTDIR itself *ends* with a dirsep,
the fact is that
   /my/destdir/C:/mingw
is an invalid path on win32 ANYWAY (e.g. if $build=win32, since $build
is where 'make install' happens), because ':' is not a legal character
for a filename!

But these issues are not something either libtool or automake should
worry about, unless they want to redesign the entire concept of DESTDIR.
(And good luck with that, when --prefix=/entirely/new/tmp/prefix already
works!)

--
Chuck