Re: [PATCH] tests: ignore minor 'recheck' regression for BSD make

[Dave Hart]

On Fri, Jul 6, 2012 at 6:02 PM, Stefano Lattarini
<address@hidden> wrote:
> It turns out that, with NetBSD 5.1 make and FreeBSD 9 make, running
> "make recheck" two times in a row quickly fails to correctly re-run
> the failed tests in the second run.
>
> That issue has been introduced likely introduced in commit

That issue was likely introduced in...

> 'v1.12.1-95-gd5443e4' of 20102-07-01, "parallel-tests: reimplement
> fix for bug#11791".
>
> Anyway, the use case that has been broken is not realistic (who is
> going to run "make recheck" two times in one second, without modifying
> any of the tests or the tested programs in the meantime?), so we believe
> the best fix is to simply work around the issue in the affected test
> cases, rather than risking to slow down or uglify the 'recheck' rule.

rather than risk slowing down or uglifying the 'recheck' rule.

> * t/parallel-tests9.sh: Enhance a little.
> * t/tap-recheck.sh: Adjust adding proper '$sleep' calls were required.

s/were/where/

> * t/parallel-tests-log-override-recheck.sh: Likewise.
> * t/test-driver-custom-multitest-recheck.sh: Likewise.
> * t/test-driver-custom-multitest-recheck2.sh: Likewise.
>
> Signed-off-by: Stefano Lattarini <address@hidden>
> ---
>
>  Please scrap the previous patch.  I don't know what I was thinking when
>  I wrote it (assuming I was thinking at all, which I much doubt).  Of
>  course it solves the regression, but doing so re-introduces bug#11791!
>  At least I did proper testing before pushing, so the error was caught
>  in time.
>
>  So, OK to apply?  I will push shortly if there is no objection.
>
>  Thanks, and sorry for the noise,
>    Stefano
>
>  t/parallel-tests-log-override-recheck.sh   |    2 ++
>  t/parallel-tests9.sh                       |    5 +++--
>  t/tap-recheck.sh                           |    7 +++++++
>  t/test-driver-custom-multitest-recheck.sh  |    8 ++++++++
>  t/test-driver-custom-multitest-recheck2.sh |    6 ++++++
>  5 files changed, 26 insertions(+), 2 deletions(-)
>
> diff --git a/t/parallel-tests-log-override-recheck.sh 
> b/t/parallel-tests-log-override-recheck.sh
> index 493fbfe..79c16fc 100755
> --- a/t/parallel-tests-log-override-recheck.sh
> +++ b/t/parallel-tests-log-override-recheck.sh
> @@ -77,6 +77,8 @@ for x in stdout my.log; do
>    $FGREP bazbaz $x
>  done
>
> +using_gmake || $sleep # Required by BSD make.
> +
>  chmod a-rw my.log
>  BAZ_EXIT_STATUS=0 TEST_SUITE_LOG=my2.log $MAKE -e recheck >stdout \
>    && { cat stdout; exit 1; }
> diff --git a/t/parallel-tests9.sh b/t/parallel-tests9.sh
> index 08d2bf6..ea9dba6 100755
> --- a/t/parallel-tests9.sh
> +++ b/t/parallel-tests9.sh
> @@ -59,9 +59,10 @@ $MAKE check >stdout && { cat stdout; exit 1; }
>  cat stdout
>  count_test_results total=3 pass=1 fail=1 skip=0 xfail=0 xpass=0 error=1
>
> -# Running this two times in a row proved problematic on NetBSD make
> -# in the past.  Ensure we don't regress in that respect.
> +# Running this two times in a row should produce the same results the
> +# second time.
>  for i in 1 2; do
> +  using_gmake || $sleep # Required by BSD make.
>    $MAKE recheck >stdout && { cat stdout; exit 1; }
>    cat stdout
>    count_test_results total=2 pass=0 fail=1 skip=0 xfail=0 xpass=0 error=1
> diff --git a/t/tap-recheck.sh b/t/tap-recheck.sh
> index 877f70e..3c542ba 100755
> --- a/t/tap-recheck.sh
> +++ b/t/tap-recheck.sh
> @@ -130,6 +130,7 @@ for vpath in : false; do
>    count_test_results total=9 pass=3 fail=2 xpass=1 xfail=1 skip=1 error=1
>
>    : Let us make b.test pass.
> +  using_gmake || $sleep # Required by BSD make.
>    echo OK > b.ok
>    do_recheck --fail
>    # a.test has been successful the first time, so no need to re-run it.
> @@ -141,6 +142,7 @@ for vpath in : false; do
>    count_test_results total=7 pass=2 fail=2 xpass=1 xfail=1 skip=1 error=0
>
>    : Let us make the first part of c.test pass.
> +  using_gmake || $sleep # Required by BSD make.
>    echo OK > c.pass
>    do_recheck --fail
>    test ! -e a.run
> @@ -150,6 +152,7 @@ for vpath in : false; do
>    count_test_results total=5 pass=1 fail=1 xpass=1 xfail=1 skip=1 error=0
>
>    : Let us make also the second part of c.test pass.
> +  using_gmake || $sleep # Required by BSD make.
>    echo KO > c.xfail
>    do_recheck --fail
>    test ! -e a.run
> @@ -160,6 +163,7 @@ for vpath in : false; do
>
>    : Nothing changed, so only d.test should be run.
>    for i in 1 2; do
> +    using_gmake || $sleep # Required by BSD make.
>      do_recheck --fail
>      test ! -e a.run
>      test ! -e b.run
> @@ -169,6 +173,7 @@ for vpath in : false; do
>    done
>
>    : Let us make d.test run more testcases, and experience _more_ failures.
> +  using_gmake || $sleep # Required by BSD make.
>    echo 'test_count=9' > d.count
>    unindent > d.extra <<'END'
>      echo 'ok # SKIP s'
> @@ -190,6 +195,7 @@ END
>    count_test_results total=11 pass=2 fail=4 xpass=1 xfail=0 skip=2 error=2
>
>    : Let us finally make d.test pass.
> +  using_gmake || $sleep # Required by BSD make.
>    echo 'test_count=1' > d.count
>    echo : > d.extra
>    do_recheck --pass
> @@ -200,6 +206,7 @@ END
>    count_test_results total=1 pass=0 fail=0 xpass=0 xfail=0 skip=1 error=0
>
>    : All tests have been successful or skipped, nothing should be re-run.
> +  using_gmake || $sleep # Required by BSD make.
>    do_recheck --pass
>    test ! -e a.run
>    test ! -e b.run
> diff --git a/t/test-driver-custom-multitest-recheck.sh 
> b/t/test-driver-custom-multitest-recheck.sh
> index ae31397..362abbf 100755
> --- a/t/test-driver-custom-multitest-recheck.sh
> +++ b/t/test-driver-custom-multitest-recheck.sh
> @@ -107,6 +107,7 @@ for vpath in : false; do
>    $srcdir/configure
>
>    : A "make recheck" in a clean tree should run no tests.
> +  using_gmake || $sleep # Required by BSD make.
>    do_recheck --pass
>    cat test-suite.log
>    test ! -e a.run
> @@ -131,6 +132,7 @@ for vpath in : false; do
>    count_test_results total=9 pass=3 fail=2 xpass=1 xfail=1 skip=1 error=1
>
>    : Let us make b.test pass.
> +  using_gmake || $sleep # Required by BSD make.
>    echo OK > b.ok
>    do_recheck --fail
>    # a.test has been successful the first time, so no need to re-run it.
> @@ -142,6 +144,7 @@ for vpath in : false; do
>    count_test_results total=7 pass=2 fail=2 xpass=1 xfail=1 skip=1 error=0
>
>    : Let us make the first part of c.test pass.
> +  using_gmake || $sleep # Required by BSD make.
>    echo OK > c.pass
>    do_recheck --fail
>    test ! -e a.run
> @@ -151,6 +154,7 @@ for vpath in : false; do
>    count_test_results total=5 pass=1 fail=1 xpass=1 xfail=1 skip=1 error=0
>
>    : Let us make also the second part of c.test pass.
> +  using_gmake || $sleep # Required by BSD make.
>    echo KO > c.xfail
>    do_recheck --fail
>    test ! -e a.run
> @@ -161,6 +165,7 @@ for vpath in : false; do
>
>    : Nothing changed, so only d.test should be run.
>    for i in 1 2; do
> +    using_gmake || $sleep # Required by BSD make.
>      do_recheck --fail
>      test ! -e a.run
>      test ! -e b.run
> @@ -170,6 +175,7 @@ for vpath in : false; do
>    done
>
>    : Let us make d.test run more testcases, and experience _more_ failures.
> +  using_gmake || $sleep # Required by BSD make.
>    unindent > d.extra <<'END'
>      echo SKIP: s
>      echo FAIL: f 1
> @@ -190,6 +196,7 @@ END
>    count_test_results total=11 pass=2 fail=4 xpass=1 xfail=0 skip=2 error=2
>
>    : Let us finally make d.test pass.
> +  using_gmake || $sleep # Required by BSD make.
>    echo : > d.extra
>    do_recheck --pass
>    test ! -e a.run
> @@ -199,6 +206,7 @@ END
>    count_test_results total=1 pass=0 fail=0 xpass=0 xfail=0 skip=1 error=0
>
>    : All tests have been successful or skipped, nothing should be re-run.
> +  using_gmake || $sleep # Required by BSD make.
>    do_recheck --pass
>    test ! -e a.run
>    test ! -e b.run
> diff --git a/t/test-driver-custom-multitest-recheck2.sh 
> b/t/test-driver-custom-multitest-recheck2.sh
> index 623cbad..b6f543a 100755
> --- a/t/test-driver-custom-multitest-recheck2.sh
> +++ b/t/test-driver-custom-multitest-recheck2.sh
> @@ -105,6 +105,7 @@ for vpath in : false; do
>    unset var
>
>    : a.test was successful the first time, no need to re-run it.
> +  using_gmake || $sleep # Required by BSD make.
>    env TESTS=a.test $MAKE -e recheck >stdout \
>      || { cat stdout; exit 1; }
>    cat stdout
> @@ -114,6 +115,7 @@ for vpath in : false; do
>    test ! -e c.run
>
>    : b.test failed, it should be re-run.  And make it pass this time.
> +  using_gmake || $sleep # Required by BSD make.
>    echo OK > b.ok
>    TEST_LOGS=b.log $MAKE -e recheck >stdout \
>      || { cat stdout; exit 1; }
> @@ -126,6 +128,7 @@ for vpath in : false; do
>    rm -f *.run
>
>    : No need to re-run a.test or b.test anymore.
> +  using_gmake || $sleep # Required by BSD make.
>    TEST_LOGS=b.log $MAKE -e recheck >stdout \
>      || { cat stdout; exit 1; }
>    cat stdout
> @@ -133,6 +136,7 @@ for vpath in : false; do
>    test ! -e a.run
>    test ! -e b.run
>    test ! -e c.run
> +  using_gmake || $sleep # Required by BSD make.
>    TESTS='a.test b.test' $MAKE -e recheck >stdout \
>      || { cat stdout; exit 1; }
>    cat stdout
> @@ -158,6 +162,7 @@ for vpath in : false; do
>    : c.test contained and hard error the last time, so it should be re-run.
>    : This time, make it pass
>    # Use 'echo', not ':'; see comments above for why.
> +  using_gmake || $sleep # Required by BSD make.
>    echo dummy > c.ok
>    env TESTS='c.test a.test' $MAKE -e recheck >stdout \
>      || { cat stdout; exit 1; }
> @@ -171,6 +176,7 @@ for vpath in : false; do
>
>    : Nothing should be rerun anymore, as all tests have been eventually
>    : successful.
> +  using_gmake || $sleep # Required by BSD make.
>    $MAKE recheck >stdout || { cat stdout; exit 1; }
>    cat stdout
>    count_test_results total=0 pass=0 fail=0 xpass=0 xfail=0 skip=0 error=0
> --
> 1.7.9.5
>
>

Cheers,
Dave Hart