Re: [PATCH] {master} Extend and improve tests on DejaGnu support.

[Stefano Lattarini]

On Thursday 09 December 2010, Ralf Wildenhues wrote:
> Hi Stefano,
> 
> * Stefano Lattarini wrote on Sun, Nov 14, 2010 at 02:29:33PM CET:
> > Another testsuite patch, based off of maint, but to be merged
> > in master only.
> 
> OK, but I have a question and a nit below.
>

> Thanks,
> Ralf
> 
> > Extend and improve tests on DejaGnu support.
> > 
> > * tests/dejagnu.test: Do not create useless dummy test script.
> > Add trailing `:' command.  In heading comments, add reference
> > to ...
> > * tests/check12.test: ... this new "semantic" test, covering
> > concurrent use of dejagnu tests, simple tests and `check-local'
> > target.
> > * tests/dejagnu2.test: Make test more reliable, by avoid weak
> > grepping of make output.  Prefer `cat' over `echo' to append
> > to configure.in.  Quote literal dots in grep regexps.  Prefer
> > `grep -c ...' over `grep ... | wc -l'.  Make grepping of
> > automake stderr slighty stricter.  Add trailing `:' command.
> 
> slightly
> 
Fixed.

> 
> > --- /dev/null
> > +++ b/tests/check12.test
> > @@ -0,0 +1,201 @@
> 
> > +# Test to make sure dejagnu tests, automake-style tests, and check-local
> > +# target can coexist.
> [...]
> > +## Go with the testcase execution.
> > +
> > +$ACLOCAL
> > +$AUTOCONF
> > +$AUTOMAKE --add-missing
> > +
> > +for vpath in : false; do
> > +
> > +  if $vpath; then
> > +    srcdir=..
> > +    mkdir build_
> > +    cd build_
> > +  else
> > +    srcdir=.
> > +  fi
> > +
> > +  $srcdir/configure
> > +
> > +  $MAKE check
> > +  test -f hammer.log
> > +  test -f hammer.sum
> > +  test -f spanner.log
> > +  test -f spanner.sum
> > +  # This checks will be run only by the autogenerated `check12-p.test'.
> > +  if test x"$parallel_tests" = x"yes"; then
> > +    cat test-suite.log
> > +    grep '^PASS: a\.test (exit: 0)' a.log
> > +    grep '^PASS: b\.test (exit: 0)' b.log
> > +  else
> > +    :
> > +  fi
> > +  grep 'check-local succeded :-)' local.log
> > +
> > +  cp -f config.status config-status.sav
> > +
> > +  $MAKE distclean
> > +  test ! -r hammer.log
> > +  test ! -r hammer.sum
> > +  test ! -r spanner.log
> > +  test ! -r spanner.sum
> > +  test ! -r test-suite.log
> > +  test ! -r a.log
> > +  test ! -r b.log
> > +  test ! -r local.log
> > +
> > +  mv -f config-status.sav config.status
> > +  ./config.status
> > +
> > +  NAIL=screw $MAKE check && Exit 1
> > +  test -f hammer.log
> > +  test -f hammer.sum
> > +  test -f spanner.log
> > +  test -f spanner.sum
> > +  grep 'FAIL: test_hammer' hammer.sum
> > +  grep 'FAIL:' spanner.sum && Exit 1
> > +
> > +  B_EXIT_STATUS=1 $MAKE check && Exit 1
> > +  # This checks will be run only by the autogenerated `check12-p.test'.
> > +  if test x"$parallel_tests" = x"yes"; then
> > +    cat test-suite.log
> > +    grep '^PASS: a\.test (exit: 0)' a.log
> > +    grep '^FAIL: b\.test (exit: 1)' b.log
> > +    grep '^FAIL: b\.test (exit: 1)' test-suite.log
> > +  else :; fi
> > +
> > +  CHECKLOCAL_EXIT_STATUS=1 $MAKE check && Exit 1
> > +  grep 'check-local failed :-(' local.log
> > +
> > +  # Do not trust the exit status of `make -k'.
> > +  NAIL=screw B_EXIT_STATUS=23 CHECKLOCAL_EXIT_STATUS=1 $MAKE -k check || :
> > +  test -f hammer.log
> > +  test -f hammer.sum
> > +  test -f spanner.log
> > +  test -f spanner.sum
> > +  grep 'FAIL: test_hammer' hammer.sum
> > +  grep 'FAIL:' spanner.sum && Exit 1
> > +  # This checks will be run only by the autogenerated `check12-p.test'.
> > +  if test x"$parallel_tests" = x"yes"; then
> > +    cat test-suite.log
> > +    grep '^PASS: a\.test (exit: 0)' a.log
> > +    grep '^FAIL: b\.test (exit: 23)' b.log
> > +    grep '^FAIL: b\.test (exit: 23)' test-suite.log
> > +  else :; fi
> > +  grep 'check-local failed :-(' local.log
> > +
> > +  $MAKE distcheck
> 
> Can't you either omit this distcheck, or the vpath = : case above,
> without decreasing coverage?
> 
The "vpath = :" case also checks the reaction to failing tests in a
VPATH build, so I'd rather not remove it.  As for the distcheck, I
think it has a real, albeit limited, usefulness, since it shows that
the package is self-contained and its distribution works as expected.
So I'd prefer to keep it, but I can also remove it if you insist.

Hmm... but thinking about it, running "make distcheck" only once would
IMHO be better in terms of performance, and wouldn't hinder coverage.

So what about the following squash-in?

-*-*-*-

diff --git a/tests/check12.test b/tests/check12.test
index 1f21a77..caa8e0f 100755
--- a/tests/check12.test
+++ b/tests/check12.test
@@ -123,6 +123,10 @@ for vpath in : false; do
     srcdir=.
   fi
 
+  if test -f config.status; then
+    $MAKE distclean
+  fi
+
   $srcdir/configure
 
   $MAKE check
@@ -192,9 +196,10 @@ for vpath in : false; do
   else :; fi
   grep 'check-local failed :-(' local.log
 
-  $MAKE distclean
   cd $srcdir
 
 done
 
+$MAKE distcheck
+
 :

-*-*-*-

Thanks,
  Stefano