Re: [PATCH] [cygwin|mingw]: Add cross-compile support to cwrapper (take 6)

[Charles Wilson]

On 8/26/2010 4:18 PM, Ralf Wildenhues wrote:
> Then, please just move the new functions where Peter needs them,
> if they really need moving, that is.

I deliberately placed them after func_compile and before func_link, for 
speed-of-parsing reasons. Obviously libtool is used to compile object 
code much more often than it is used to link (since every linked result 
requires one or more, sometimes many more, objects) -- so moving these 
functions ahead of func_compile will impact speed.  How much? Don't 
know; I'll try to generate some numbers.

OTOH, it is absolutely *required* to move them where Peter wants them, 
since he /must/ use translate some paths if func_compile is to work, 
with MSVC.  So...we have to pay the price regardless.

Also: I've said this before, but we can't use the m4 function_replace 
magic because we need to retain the ability for users to override the 
choice of $func_to_host_path_cmd.  This is critical for the 
'cygwin->mingw (lie)' use case, which is VERY common for the gcc developers.

> Then, since the next few days might see a number of commits: in case
> your redone patch does not apply or rebase cleanly against master any
> more, you can easily leave that rebasing work to me, or better, just
> apply it to the tree you tested against and let's merge that from
> master (again, I can do that for you).

No worries. I have no issues with rebasing...but I don't plan on redoing 
a week's worth of test suite runs afterward.  Maybe a day's worth... :-)

> * Charles Wilson wrote on Mon, Jul 19, 2010 at 03:07:01AM CEST:
> > * libltdl/m4/libtool.m4 (_LT_PATH_CONVERSION_FUNCTIONS): New
> > function sets libtool variable $to_host_path_cmd, and employs
> > cache. AC_SUBSTs $to_host_path_cmd, as well.
> > (_LT_SETUP): Require it.
> > * tests/testsuite.at: Ensure to_host_path_cmd is passed as a
> > variable setting on the configure line for (new testsuite) tests.
> > * Makefile.am: Ensure to_host_path_cmd is included in
> > TEST_ENVIRONMENT so that it is passed to (old testsuite) tests.
>
> Typo TESTS_ENVIRONMENT.

Ack.

> > * libltdl/config/ltmain.m4sh (func_cygpath): New function.
> > (func_init_to_host_pathlist_cmd): New function.
> > (func_to_host_path): Refactored to... (now uses eval
> > $to_host_path_cmd).
> > (func_wine_to_win32_path): Here. New function.
> > (func_msys_to_win32): Here. New function.
> > (func_path_convert_check): Here. New function.
> > (func_noop_path_convert): Here. New function.
> > (func_msys_to_mingw_path_convert): Here. New function.
> > (func_cygwin_to_mingw_path_convert): Here. New function.
> > (func_nix_to_mingw_path_convert): Here. New function.
> > (func_msys_to_cygwin_path_convert): New function.
> > (func_nix_to_cygwin_path_convert): New function.
> > (func_to_host_pathlist): Refactored to... (now uses eval
> > $to_host_pathlist_cmd and func_init_to_host_pathlist_cmd).
> > (func_pathlist_convert_check): Here. New function.
> > (func_pathlist_front_back_pathsep): Here. New function.
> > (func_wine_to_win32_pathlist): Here. New function.
> > (func_noop_pathlist_convert): Here. New function.
> > (func_msys_to_mingw_pathlist_convert): Here. New function.
> > (func_cygwin_to_mingw_pathlist_convert): Here. New function.
> > (func_nix_to_mingw_pathlist_convert): Here. New function.
> > (func_msys_to_cygwin_pathlist_convert): New function.
> > (func_nix_to_cygwin_pathlist_convert): New function.
>
> I'm not sure I've asked before, but will state again: coding up X-to-Y
> for N choices of X and M choices of Y has complexity N*M, while coding
> it up as from-X and to-Y has complexity N+M.  Quadratic algorithms don't
> scale.  Why is the latter not done?

I don't think it has come up explicitly, but it also occurred 
independently to me.  The main issue is you don't know what the "native" 
(e.g. "central") path-type is; e.g. "from-X (to what?)" and "(from 
what?) to-Y".

Right now there are only four "platforms" involved: *nix, mingw, msys, 
and cgywin.  That's actually the break-even point, given the vagaries 
and optimizations involved in these particular four platforms.

We have exactly five basic _convert functions (not counting the 
_pathlist_convert extensions).

For a from-X|to-Y solution, we'd need, I think, the same number of 
_convert functions: brute force suggests nine (four to_*, four from_*, 
plus the noop), but then many of the from_* and to_* would actually BE noop.

I'm assuming here that the "central" path-type is implicitly some sort 
of unixish -- maybe cyg, maybe msys, maybe unix -- path-type.  The issue 
is that each of the five conversion functions use a different TOOL to 
perform the conversion, with different syntax.  So, trying to combine, e.g.
   msys_to_mingw
   cygwin_to_mingw
   unix_to_mingw
into an all-encompassing "central_unixish_to_mingw" would require 
additional m4 magic to basically replace the guts depending on whether 
$build was msys, cygwin, or unix.  (And you can't really do a set of 
{msys|cygwin|unix}_to_central_unixish that isn't simply a no-op -- 
because (A) they already are all unixish, and (B) what tool would you 
use? How would the later to_mingw function "know" how to covert this new 
representation to mingw, unless the 
{msys|cygwin|unix}_to_central_unixish was just a no-op and to_mingw 
actually did all the work?)

In short, I think once again we are trapped in the "geez, win32 is 
freaking bizarre!" maze.

For more reasonable cross environments (e.g. linux->some_embedded) I 
think you could probably work out a general M+N scheme, since most 
embedded $hosts aren't as strange as the win32 variants -- even VxWorks 
and INTEGRITY have basic, unix-like file systems (although INTEGRITY 
does have multiple roots).  (And, aggressive use of the m4 
function_replace machinery WOULD be appropriate for /these/ conversion 
functions).

OTOH...(a) you can't run the $host apps on $build anyway, in that case 
-- at best you'd use $TARGETSHELL and "run" them via a remote 
connection, and (b) they don't use the C wrapper!

So...I don't think it makes much difference *right now* in the amount of 
code required, or the number of functions implemented.  OTOH, as a 
follow-on patch after 2.2.next, we could implement an explicit N+M 
scheme just so that any future extensions -- which I can't actually 
foresee -- could "hook in" scalably.

> The answer should be in a comment
> in the code, if it cannot be done.  If it can be done, then I am OK with
> making it a TODO item to be addressed after 2.2.12, rationale being that
> that's just an optimization so QoI issue, whereas your patch brings new
> features.

Err... ^^^ that's quite a long comment to parse in ltmain.m4sh...maybe a 
short note to see the TODO file, and put the bulk there?

> > mingw-native (e.g. msys->mingw)
>
> This is interesting.  I don't see these mdemo*exec failures in my
> mingw-native setup.  I wonder why that is.  When parallel-tests is
> merged, we should look at test-suite.log.

I still have all the separate build envs. I can take a look, the "old 
way" even without parallel-tests support.  Will post later.

> > cygwin->mingw cross (faked)
> > ===================
> > This is where you do:
> >     $ export PATH="/c/MinGW/bin:${PATH}"
> >     $ configure --build=i686-pc-cygwin --host=mingw32 \
> >         NM=/c/MinGW/bin/nm.exe
> > That is, you use cygwin as $build, but use the native MinGW
> > compiler (instead of a cygwin-hosted cross compiler).
> >
> > I haven't tested this yet (I forgot about it) but I really should.
> > It is a common use case.
>
> Yes, this would be interesting to see.

Unfortunately, it doesn't work. The build itself -- never mind the tests 
-- fails due to the absolute-pathing vs. ar issue, that Peter has 
mentioned. (ar is the MinGW one, and doesn't grok an absolute cygwin path).

libtool: link: (cd libltdl/.libs/libltdl.lax/loadlibrary.a && ar x 
"/usr/src/packages/libtool/git/build-cygwin-mingw-cross-fake-wine-support/libltdl/.libs/loadlibrary.a")
/usr/src/packages/libtool/git/build-cygwin-mingw-cross-fake-wine-support/libltdl/.libs/loadlibrary.a: 
No such file or directory
C:\MinGW\bin\ar.exe: make[2]: *** [libltdl/libltdl.la] Error 9

So, testing this use case will actually have to wait until after both my 
patch and (some of) Peter's are merged.  However, I think this use case 
is *currently* broken in master anyway, so it's not a regression.


> > cygwin->mingw cross (lie)
> > ===================
[snip limitations]
> > I haven't tested this recently, but I need to. I just forgot.
>
> All these limitations seem acceptable to me for this particular case.

You also must have "identity mounts". I don't anymore, so I'll have to 
set that up on an alternate computer or VM.

> > linux->mingw cross
> > ==================
> > linux->mingw (old tests): 2 of 100 FAIL, 6 SKIP
> >    FAIL: tests/demo-hardcode.test
> >    FAIL: tests/depdemo-relink.test
> >    Don't know if these are regressions or not; will recheck without
> >    this patch.
>
> Whether or not, it'd be interesting to see verbose logs of these at some
> point.

They aren't. I have the logs; will post separately later.

> > linux->cygwin cross (LT_CYGPATH not set)
> > ===================
> > linux->cygwin (old tests): 1 of 114 FAIL, 10 SKIP
> >    FAIL: tests/demo-hardcode.test
> >    Ditto: don't know if this is a regression or not; will recheck
> >    without this patch.
>
> Ditto.

See below.

> > linux->cygwin (new tests): AWFUL.
> >                            23 unexpected failures
> >                            32 skip
> > I don't expect any difference WITH LT_CYGPATH set, because
> > cygpath-1.7 (and, indeed, all cygwin-1.7 programs) seems to crash under
> > wine anyway.
>
> OK, so we have to ignore results from this configuration as long as that
> issue is not fixed.

Well, as it turns out (see my other message in this thread) it appears 
my linux->cygwin cross compiler is borked.  So, I need to re-do THAT and 
make sure it is working first.

That doesn't change the fact that cgywin-1.7+wine == fail. So... :-(

I've got to run, so: I promise at least two more replies to your review. 
(1) point-by-point with respect to the code comments, and (2) the 
promised verbose logs from selected tests above.

> To let you not feel that I am making this another neverending story:
> please, after revising, if you don't have open questions, do commit the
> patch, judging yourself whether it is suitable for master yet or, if you
> deem it not, just push it to some new branch, let's call it
> file-conversion for lack of a better name.

Ack.

--
Chuck