Re: [PATCH] {master} Extend and improve tests on DejaGnu support.

[Ralf Wildenhues]

Hi Stefano,

* Stefano Lattarini wrote on Sun, Nov 14, 2010 at 02:29:33PM CET:
> Another testsuite patch, based off of maint, but to be merged
> in master only.

OK, but I have a question and a nit below.

Thanks,
Ralf

> Extend and improve tests on DejaGnu support.
> 
> * tests/dejagnu.test: Do not create useless dummy test script.
> Add trailing `:' command.  In heading comments, add reference
> to ...
> * tests/check12.test: ... this new "semantic" test, covering
> concurrent use of dejagnu tests, simple tests and `check-local'
> target.
> * tests/dejagnu2.test: Make test more reliable, by avoid weak
> grepping of make output.  Prefer `cat' over `echo' to append
> to configure.in.  Quote literal dots in grep regexps.  Prefer
> `grep -c ...' over `grep ... | wc -l'.  Make grepping of
> automake stderr slighty stricter.  Add trailing `:' command.

slightly

> * tests/dejagnu3.test: Prefer `cat' over `echo' to append to
> configure.in.  Check stderr of expected-to-fail "make" call.
> Remove extra blank lines from Makefile.am.
> * tests/dejagnu4.test: Prefer `cat' over `echo' to append to
> configure.in.  Prefer `mv -f' over plain `mv' when the target
> file already exists.  Avoid extra mkdir calls by creating more
> directories at once.  Better use of blank lines.  Check that
> the `*.log' and `*.sum' files are created by runtest also when
> "make check" fails.
> * tests/dejagnu7.test: Prefer `cat' over `echo' to append to
> configure.in.  Better use of blank lines.  Add a trailing `:'
> command.
> * tests/dejagnu6.test: Likewise, and give the dejagnu test a
> more descriptive name.
> * tests/dejagnu5.test: Likewise.  Also, simply define package
> name to `$me' rather than using a non-obvious sed script to
> extract it from `AC_INIT', and write the Makefile.am with only
> one command.
> * tests/Makefile.am (TESTS): Updated.

> --- /dev/null
> +++ b/tests/check12.test
> @@ -0,0 +1,201 @@

> +# Test to make sure dejagnu tests, automake-style tests, and check-local
> +# target can coexist.
[...]
> +## Go with the testcase execution.
> +
> +$ACLOCAL
> +$AUTOCONF
> +$AUTOMAKE --add-missing
> +
> +for vpath in : false; do
> +
> +  if $vpath; then
> +    srcdir=..
> +    mkdir build_
> +    cd build_
> +  else
> +    srcdir=.
> +  fi
> +
> +  $srcdir/configure
> +
> +  $MAKE check
> +  test -f hammer.log
> +  test -f hammer.sum
> +  test -f spanner.log
> +  test -f spanner.sum
> +  # This checks will be run only by the autogenerated `check12-p.test'.
> +  if test x"$parallel_tests" = x"yes"; then
> +    cat test-suite.log
> +    grep '^PASS: a\.test (exit: 0)' a.log
> +    grep '^PASS: b\.test (exit: 0)' b.log
> +  else
> +    :
> +  fi
> +  grep 'check-local succeded :-)' local.log
> +
> +  cp -f config.status config-status.sav
> +
> +  $MAKE distclean
> +  test ! -r hammer.log
> +  test ! -r hammer.sum
> +  test ! -r spanner.log
> +  test ! -r spanner.sum
> +  test ! -r test-suite.log
> +  test ! -r a.log
> +  test ! -r b.log
> +  test ! -r local.log
> +
> +  mv -f config-status.sav config.status
> +  ./config.status
> +
> +  NAIL=screw $MAKE check && Exit 1
> +  test -f hammer.log
> +  test -f hammer.sum
> +  test -f spanner.log
> +  test -f spanner.sum
> +  grep 'FAIL: test_hammer' hammer.sum
> +  grep 'FAIL:' spanner.sum && Exit 1
> +
> +  B_EXIT_STATUS=1 $MAKE check && Exit 1
> +  # This checks will be run only by the autogenerated `check12-p.test'.
> +  if test x"$parallel_tests" = x"yes"; then
> +    cat test-suite.log
> +    grep '^PASS: a\.test (exit: 0)' a.log
> +    grep '^FAIL: b\.test (exit: 1)' b.log
> +    grep '^FAIL: b\.test (exit: 1)' test-suite.log
> +  else :; fi
> +
> +  CHECKLOCAL_EXIT_STATUS=1 $MAKE check && Exit 1
> +  grep 'check-local failed :-(' local.log
> +
> +  # Do not trust the exit status of `make -k'.
> +  NAIL=screw B_EXIT_STATUS=23 CHECKLOCAL_EXIT_STATUS=1 $MAKE -k check || :
> +  test -f hammer.log
> +  test -f hammer.sum
> +  test -f spanner.log
> +  test -f spanner.sum
> +  grep 'FAIL: test_hammer' hammer.sum
> +  grep 'FAIL:' spanner.sum && Exit 1
> +  # This checks will be run only by the autogenerated `check12-p.test'.
> +  if test x"$parallel_tests" = x"yes"; then
> +    cat test-suite.log
> +    grep '^PASS: a\.test (exit: 0)' a.log
> +    grep '^FAIL: b\.test (exit: 23)' b.log
> +    grep '^FAIL: b\.test (exit: 23)' test-suite.log
> +  else :; fi
> +  grep 'check-local failed :-(' local.log
> +
> +  $MAKE distcheck

Can't you either omit this distcheck, or the vpath = : case above,
without decreasing coverage?

> +  $MAKE distclean
> +  cd $srcdir
> +
> +done
> +
> +: