Re: Spurious tests are run in the old testsuite.

[Peter Rosin]

Den 2010-09-09 14:04 skrev Peter Rosin:
> When I run this on a freshly bootstrapped, configured and made tree:
> 
> make check-TESTS TESTS="tests/mdemo-conf.test tests/mdemo-make.test 
> tests/mdemo-exec.test"
> 
> I get this output on Cygwin:
> 
> make[1]: Entering directory `/home/peda/libtool/git/cygwin'
> PASS: tests/mdemo-static.test
> PASS: tests/mdemo-static-make.test
> PASS: tests/mdemo-static-exec.test
> PASS: tests/mdemo-static-inst.test
> PASS: tests/mdemo-static-unst.test
> PASS: tests/mdemo-conf.test
> PASS: tests/mdemo-make.test
> PASS: tests/mdemo-exec.test
> ==================
> All 3 tests passed
> ==================
> make[1]: Leaving directory `/home/peda/libtool/git/cygwin'
> 
> So, it runs 8 tests but only counts the requested 3 three.
> 
> I do get the expected behavior if I issue the command a second time.
> I.e.
> 
> make[1]: Entering directory `/home/peda/libtool/git/cygwin'
> PASS: tests/mdemo-conf.test
> PASS: tests/mdemo-make.test
> PASS: tests/mdemo-exec.test
> ==================
> All 3 tests passed
> ==================
> make[1]: Leaving directory `/home/peda/libtool/git/cygwin'
> 
> 
> And I do get the expected behavior on MSYS/MinGW using the same
> bootstrap, but configured and made in another directory. But maybe
> that is related to me having run the Cygwin tests first?

I have verified that is indeed related to me running the Cygwin
tests first.

> I suspect that some new dependency information needed by the
> parallel tests changes leaks in at the wrong place.

This is very annoying. I originally had this:

...
PASS: tests/mdemo-conf.test
PASS: tests/mdemo-make.test
PASS: tests/mdemo-exec.test
PASS: tests/mdemo-inst.test
PASS: tests/mdemo-unst.test
FAIL: tests/mdemo-dryrun.test
PASS: tests/mdemo2-conf.test
SKIP: tests/mdemo2-make.test
SKIP: tests/mdemo2-exec.test
...

and noticed that the failing mdemo-dryrun.test causes the skips in mdemo2,
so I tried to not run mdemo-dryrun.test like this:

VERBOSE=yes make check-TESTS TESTSUITEFLAGS=-V TESTS="tests/mdemo-conf.test 
tests/mdemo-make.test tests/mdemo-exec.test"

with the following outout:

make[1]: Entering directory `/home/peda/libtool/git/cygwin'
PASS: tests/mdemo-conf.test
PASS: tests/mdemo-make.test
PASS: tests/mdemo-exec.test
==================
All 3 tests passed
==================
make[1]: Leaving directory `/home/peda/libtool/git/cygwin'

Cool. I followed up with:

VERBOSE=yes make check-TESTS TESTSUITEFLAGS=-V TESTS="tests/mdemo2-conf.test 
tests/mdemo2-make.test tests/mdemo2-exec.test"

with this output:

make[1]: Entering directory `/home/peda/libtool/git/cygwin'
PASS: tests/mdemo2-conf.test
PASS: tests/mdemo-inst.test
PASS: tests/mdemo-unst.test
FAIL: tests/mdemo-dryrun.test
SKIP: tests/mdemo2-make.test
SKIP: tests/mdemo2-exec.test
======================
1 test passed
(2 tests were not run)
======================
make[1]: Leaving directory `/home/peda/libtool/git/cygwin'

Sigh. So, there is no way to run mdemo2-make.test and mdemo2-exec.test
on systems where mdemo-dryrun.test fails.

Cheers,
Peter