Testsuite issues (was: [PATCH 6/7] Convert file name to toolchain format when invoking $NM.)

[Peter Rosin]

Den 2010-09-10 07:38 skrev Ralf Wildenhues:
> Hi Peter,
> 
> * Peter Rosin wrote on Fri, Sep 10, 2010 at 12:35:59AM CEST:
>> Den 2010-09-09 22:05 skrev Charles Wilson:
>>> On 9/9/2010 3:56 PM, Ralf Wildenhues wrote:
>>>> Secondly, I can help with unsupervised git bisect if you need.  In
>>>> <http://lists.gnu.org/archive/html/bug-libtool/2010-09/msg00006.html>,
> 
>> I adjusted both reconfdirs and TESTS. You tend to learn how to do that
>> quickly in windows-land. The payoff for that learning curve is almost
>> instantaneous.
> 
> Hehe.
> 
>>> I'll take a look, but it may actually be faster to do it old-school:
>>> just trace thru the failing case in git master-NOW, figure out what's
>>> going wrong, and fix it.
>>
>> In my current setup it's hard to automate, as I do git and bootstrap on
>> Cygwin only, then test Cygwin and when that is done I switch to MSYS and
>> test there too.
> 
> Ok, so the payoff for changing that will be noticeable then, too.
> Install msysgit (someplace else), make sure the git program is found
> in PATH in your normal MSYS installation.  Profit.

Well, only until you mess things up and you don't know what you have
done. Running autotools from both left and right seems like a sure
way to create a mess. And when I do foul things up, I would have to go
back from the top and that's never fun. Also, I don't trust MSYS to not
add dos style line endings somewhere, or to not foul up the permissions
so that I accidentally start committing executable source files and what
not. I just don't feel comfortable with that.

But the above is not related to running a git bisect in general. 
Installing and setting up msysgit and current autotools and getting
things to work would not have been faster than just doing it the way
I did. At least I don't think so. Not for me anyway, remember that
I'm not familiar with running various versions of these tools, and
that I don't know exactly how they interact and how to control
wrappers etc. I could easily have spent hours barking up the wrong
tree. Anyway, the short term benefit was not obvious.

>> I have version mismatches for automake and autoconf in
>> MSYS and Cygwin, so if I'm not careful things go belly up. I should
>> perhaps try to have matching autotools, but I'm used to not having them
>> at all in MSYS, and I'm not sure if things will be screwed up with paths
>> leaking in from the other posix world if I mix, so I'm just doing it in
>> a way that I know works.
> 
> Sure.  I have current autotools in $HOME/local in MSYS, put that early
> in PATH, profit (also because it executes a bit faster than Cygwin).
> 
>> Watch out, rant coming up. You have been warned.
>>
>> My problem(?) is that libltdl issues are the hardest to fix, so there
>> are multiple issues that are not fixed for MSVC. That, and that most
>> libltdl tests are too wide for my liking, so the libltdl part of the
>> testsuite tend to either be all failing or all failing even if you have
>> made an improvement. Which makes it hard to use the testsuite to
>> determine if you have made any progress when you are testing something.
> 
> Fully agreed.  I already mentioned that improving ltdl coverage would be
> paramount for any rewriting in that area, this is just another reason.
> 
>> It is also hard to deduce if you have caused some spectacular new fail
>> with a patch, if the test in question fails before it got to the newly
>> introduced badness.
> 
> So are you saying we should have compile+link-only ltdl tests?
> Which ones are likely to fail early?

I honestly don't know. It's probably not fixable with only testsuite
work. When looking up what goes belly up with libltdl, you will almost
always need to dig into the C code, or in some symbol crap that's too
low level for the testsuite to be able to reasonably know anything
about. Unless the testsuite is taught internals such as the expected
content of the *S.c file in various circumstances etc, but creating
and maintaining such tests would be a nightmare.

To test various parts of the C code of libltdl without doing a full
test (that would only be able to say "fail"), you would probably need
to add some tracing mechanism to see that the expected code paths are
taken. But the problem is of course that this is mainly needed when
porting to alien stuff like Windows, and in those alien cases the
normally expected code paths are perhaps not always taken for various
other reasons, voiding the whole scheme.

Regarding other stuff, like the hardship with the stresstest.at
looping galore, I think that instead of having the current loops, it
would be useful to run all 2*3*2*3*3*2 variations regardless, even if
one variation failed in the beginning. If the termination of the test
could be delayed, and the result of the LT_AT_{EXEC_,}CHECK macros
could be recorded and shown at the end of the stresstest log, I think
that would have been extremely useful.

IOW, a table something like this instead of only fail:282 (and the
inevitable hunt for which 282 that would be, it's much better with
the eval at line 263, but still not obvious):

linkargs="-no-undefined  "

    rel    st          selfsyms                     result
1                                                   ok
2                      -export-symbols dlselfsyms   ok
3          -static                                  ok
4          -static     -export-symbols dlselfsyms   ok
5          -no-install                              ok
6          -no-install -export-symbols dlselfsyms   ok
7   ./                                              ok
8   ./                 -export-symbols dlselfsyms   ok
9   ./     -static                                  ok
10  ./     -static     -export-symbols dlselfsyms   ok
11  ./     -no-install                              ok
12  ./     -no-install -export-symbols dlselfsyms   ok
13  `pwd`/                                          ok
14  `pwd`/             -export-symbols dlselfsyms   ok
15  `pwd`/ -static                                  ok
16  `pwd`/ -static     -export-symbols dlselfsyms   ok
17  `pwd`/ -no-install                              ok
18  `pwd`/ -no-install -export-symbols dlselfsyms   ok

linkargs="-no-undefined  -rpath /nonexistent"

    rel    st          selfsyms                     result
19                                                  ok
20                     -export-symbols dlselfsyms   ok
21         -static                                  ok
22         -static     -export-symbols dlselfsyms   ok
23         -no-install                              ok
24         -no-install -export-symbols dlselfsyms   ok
25  ./                                              ok
26  ./                 -export-symbols dlselfsyms   ok
27  ./     -static                                  ok
28  ./     -static     -export-symbols dlselfsyms   ok
29  ./     -no-install                              ok
30  ./     -no-install -export-symbols dlselfsyms   ok
31  `pwd`/                                          ok
32  `pwd`/             -export-symbols dlselfsyms   ok
33  `pwd`/ -static                                  ok
34  `pwd`/ -static     -export-symbols dlselfsyms   ok
35  `pwd`/ -no-install                              ok
36  `pwd`/ -no-install -export-symbols dlselfsyms   ok

linkargs="-no-undefined -export-symbols-regex "v.*" "

    rel    st          selfsyms                     result
37                                                  ok
38                     -export-symbols dlselfsyms   ok
39         -static                                  ok
30         -static     -export-symbols dlselfsyms   ok
41         -no-install                              ok
42         -no-install -export-symbols dlselfsyms   fail:282
43  ./                                              fail:263
44  ./                 -export-symbols dlselfsyms   fail:263
45  ./     -static                                  fail:263
46  ./     -static     -export-symbols dlselfsyms   fail:263
47  ./     -no-install                              fail:263
48  ./     -no-install -export-symbols dlselfsyms   fail:263
49  `pwd`/                                          ok
40  `pwd`/             -export-symbols dlselfsyms   ok
51  `pwd`/ -static                                  ok
52  `pwd`/ -static     -export-symbols dlselfsyms   ok
53  `pwd`/ -no-install                              ok
54  `pwd`/ -no-install -export-symbols dlselfsyms   ok

...

I.e., one line for each run of the innermost loop. If a test leading
up to that innermost test failed, all affected entries in the
innermost loop would be marked the same way (fail:263 above).

Patterns in such a table would probably help pinpointing regressions.

I'm having fuzzy dreams of course.

>> Even if the the new fail happens before the old
>> fail, it's still harder to discriminate fail:63 from fail:67 than it is
>> to discriminate fail:63 from ok. It also doesn't really help that I'm
>> not using libltdl for anything real: I'm not really familiar with what
>> the expected behavior is and it tends to not be that important and
>> "itchy" to me.
> 
> What would help here, though?  More descriptive comments in the test
> sources?

I don't know what to do about it, I was just ranting on about my pain.
I think the fundamental problem is that I'm not using libltdl for
anything real, so nothing to scratch.

>> And then, as always, there's the wall clock issue.
> 
> Yep.  Dunno what to do about that.

Not much to do about that, no.

>> stresstest is now ok for MSVC, so it is finally really useful in
>> detecting regressions, and no longer a "masking failure rat hole".
> 
> Cool.

Yes.  I wanted to finish up with something positive. :-)

Cheers,
Peter