[PATCH 2/5] {test-protocols} parallel-tests: new recognized test result

automake-patches
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[PATCH 2/5] {test-protocols} parallel-tests: new recognized test result

From:	Stefano Lattarini
Subject:	[PATCH 2/5] {test-protocols} parallel-tests: new recognized test result 'ERROR'
Date:	Fri, 15 Jul 2011 00:37:07 +0200
User-agent:	KMail/1.13.3 (Linux/2.6.30-2-686; KDE/4.4.4; i686; ; )
* lib/am/check.am ($(TEST_SUITE_LOG)): Recognize a new test result
`ERROR'.  Use it when encountering unreadable test logs (previously
a simple `FAIL' was used in this situations).
* lib/test-driver: Set the global test result to `ERROR' when the
test exit status is 99.  When doing colorized output, color `ERROR'
results in magenta.
* doc/automake.texi (Log files generation and test results
recording): Update by also listing `ERROR' among the list of valid
`:test-results:' arguments.
* NEWS: Update.
* tests/trivial-test-driver: Update.
* tests/parallel-tests.test: Likewise.
* tests/parallel-tests-harderror.test: Likewise.
* tests/parallel-tests-no-spurious-summary.test: Likewise.
* tests/test-driver-global-log.test: Likewise.
* tests/test-driver-recheck.test: Likewise.
* tests/test-driver-custom-multitest-recheck.test: Likewise.
* tests/test-driver-custom-multitest-recheck2.test: Likewise.
* tests/test-driver-custom-multitest.test: Likewise.
* tests/test-driver-custom-no-html.test: Likewise.
* tests/test-driver-end-test-results.test: Likewise.
* tests/color.test: Likewise.  Also, make stricter, and also test
from VPATH.
* tests/color2.test: Likewise, and improve syncing with color.test.
* tests/parallel-tests-exit-statuses.test: New test.
* tests/parallel-tests-console-output.test: Likewise.
* tests/Makefile.am (TESTS): Update.
---
 ChangeLog                                        |   31 ++++++
 NEWS                                             |    4 +
 doc/automake.texi                                |    8 +-
 lib/Automake/tests/Makefile.in                   |    4 +-
 lib/am/check.am                                  |    8 +-
 lib/test-driver                                  |    5 +-
 tests/Makefile.am                                |    2 +
 tests/Makefile.in                                |    6 +-
 tests/color.test                                 |  101 +++++++++++++++-----
 tests/color2.test                                |  108 ++++++++++++++++------
 tests/parallel-tests-console-output.test         |  104 +++++++++++++++++++++
 tests/parallel-tests-exit-statuses.test          |  108 ++++++++++++++++++++++
 tests/parallel-tests-harderror.test              |    6 +-
 tests/parallel-tests-no-spurious-summary.test    |    2 +
 tests/parallel-tests.test                        |    9 +-
 tests/test-driver-custom-multitest-recheck.test  |   27 +++---
 tests/test-driver-custom-multitest-recheck2.test |   56 ++++++++---
 tests/test-driver-custom-multitest.test          |   33 ++++---
 tests/test-driver-custom-no-html.test            |    2 +-
 tests/test-driver-end-test-results.test          |    1 +
 tests/test-driver-global-log.test                |    4 +-
 tests/test-driver-recheck.test                   |    2 +-
 tests/trivial-test-driver                        |   11 +-
 23 files changed, 524 insertions(+), 118 deletions(-)
 create mode 100755 tests/parallel-tests-console-output.test
 create mode 100755 tests/parallel-tests-exit-statuses.test

diff --git a/ChangeLog b/ChangeLog
index f7da174..6a062e3 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,5 +1,36 @@
 2011-07-07  Stefano Lattarini  <address@hidden>
 
+       parallel-tests: new recognized test result 'ERROR'
+       * lib/am/check.am ($(TEST_SUITE_LOG)): Recognize a new test result
+       `ERROR'.  Use it when encountering unreadable test logs (previously
+       a simple `FAIL' was used in this situations).
+       * lib/test-driver: Set the global test result to `ERROR' when the
+       test exit status is 99.  When doing colorized output, color `ERROR'
+       results in magenta.
+       * doc/automake.texi (Log files generation and test results
+       recording): Update by listing `ERROR' too among the list of valid
+       `:test-results:' arguments.
+       * NEWS: Update.
+       * tests/trivial-test-driver: Update.
+       * tests/parallel-tests.test: Likewise.
+       * tests/parallel-tests-harderror.test: Likewise.
+       * tests/parallel-tests-no-spurious-summary.test: Likewise.
+       * tests/test-driver-global-log.test: Likewise.
+       * tests/test-driver-recheck.test: Likewise.
+       * tests/test-driver-custom-multitest-recheck.test: Likewise.
+       * tests/test-driver-custom-multitest-recheck2.test: Likewise.
+       * tests/test-driver-custom-multitest.test: Likewise.
+       * tests/test-driver-custom-no-html.test: Likewise.
+       * tests/test-driver-end-test-results.test: Likewise.
+       * tests/color.test: Likewise.  Also, make stricter, and also test
+       from VPATH.
+       * tests/color2.test: Likewise, and improve syncing with color.test.
+       * tests/parallel-tests-exit-statuses.test: New test.
+       * tests/parallel-tests-console-output.test: Likewise.
+       * tests/Makefile.am (TESTS): Update.
+
+2011-07-07  Stefano Lattarini  <address@hidden>
+
        parallel-tests: make parsing of test results safer
        The new code for parsing the testsuite-generated `.log' files,
        as introduced in commit `v1.11-872-gc96b881', considers each
diff --git a/NEWS b/NEWS
index 37af53a..56250ab 100644
--- a/NEWS
+++ b/NEWS
@@ -13,6 +13,10 @@ New in 1.11a:
 
 * Changes to Automake-generated testsuite harnesses:
 
+  - The parallel-tests harness has a new test result 'ERROR', that can be
+    used to signal e.g., unexpected or internal errors, or failures to set
+    up test case scenarios.
+
   - The default testsuite driver offered by the 'parallel-tests' option is
     now implemented (partly at least) with the help of automake-provided
     auxiliary scripts (e.g., `test-driver'), instead of relying entirely
diff --git a/doc/automake.texi b/doc/automake.texi
index 1e251db..16804cd 100644
--- a/doc/automake.texi
+++ b/doc/automake.texi
@@ -9248,10 +9248,10 @@ leading whitespace will not be ignored.
 
 @c Keep this in sync with lib/am/check-am:$(TEST_SUITE_LOG).
 The only recognized test results are currently @code{PASS}, @code{XFAIL},
address@hidden, @code{FAIL} and @code{XPASS}.  These results, when declared
-with @code{:test-result:}, can be optionally followed by text holding
-the name and/or a brief description of the corresponding test; the
address@hidden harness will ignore such extra text when
address@hidden, @code{FAIL}, @code{XPASS} and @code{ERROR}.  These results,
+when declared with @code{:test-result:}, can be optionally followed by
+text holding the name and/or a brief description of the corresponding
+test; the @option{parallel-tests} harness will ignore such extra text when
 generating @file{test-suite.log} and preparing the testsuite summary.
 Also, @code{:test-result:} can be used with a special ``pseudo-result''
 @code{END}, that will instruct the testsuite harness to stop scanning
diff --git a/lib/Automake/tests/Makefile.in b/lib/Automake/tests/Makefile.in
index 5d3c39f..82ef9e1 100644
--- a/lib/Automake/tests/Makefile.in
+++ b/lib/Automake/tests/Makefile.in
@@ -348,7 +348,7 @@ $(TEST_SUITE_LOG): $(TEST_LOGS)
        nlinit=`echo 'nl="'; echo '"'`; eval "$$nlinit"; unset nlinit; \
        list='$(TEST_LOGS)'; \
        list2=`for f in $$list; do test ! -r $$f || echo $$f; done`; \
-       results1=`for f in $$list; do test -r $$f || echo FAIL; done`; \
+       results1=`for f in $$list; do test -r $$f || echo ERROR; done`; \
        results2=''; \
        exec 5<&0; \
        for f in $$list2; do \
@@ -369,6 +369,8 @@ $(TEST_SUITE_LOG): $(TEST_LOGS)
        skip=`echo "$$results" | grep -c '^SKIP'`;                      \
        xfail=`echo "$$results" | grep -c '^XFAIL'`;                    \
        xpass=`echo "$$results" | grep -c '^XPASS'`;                    \
+       error=`echo "$$results" | grep -c '^ERROR'`;                    \
+       fail=`expr $$fail + $$error`;                                   \
        failures=`expr $$fail + $$xpass`;                               \
        all=`expr $$all - $$skip`;                                      \
        if test "$$all" -eq 1; then tests=test; All=;                   \
diff --git a/lib/am/check.am b/lib/am/check.am
index f2528e0..3cedd46 100644
--- a/lib/am/check.am
+++ b/lib/am/check.am
@@ -66,7 +66,7 @@ include inst-vars.am
 ## appended.
 ##
 ## In addition to the magic "exit 77 means SKIP" feature (which was
-## imported from automake), there is a magic "exit 99 means FAIL" feature
+## imported from automake), there is a magic "exit 99 means ERROR" feature
 ## which is useful if you need to issue a hard error no matter whether the
 ## test is XFAIL or not.  You can disable this feature by setting the
 ## variable DISABLE_HARD_ERRORS to a nonempty value.
@@ -153,7 +153,7 @@ $(TEST_SUITE_LOG): $(TEST_LOGS)
 ## Readable test logs.
        list2=`for f in $$list; do test ! -r $$f || echo $$f; done`; \
 ## Each unreadable test log counts as a failed test.
-       results1=`for f in $$list; do test -r $$f || echo FAIL; done`; \
+       results1=`for f in $$list; do test -r $$f || echo ERROR; done`; \
 ## Now we're going to extract the outcome of all the testcases from the
 ## test logs.
        results2=''; \
@@ -187,6 +187,10 @@ $(TEST_SUITE_LOG): $(TEST_LOGS)
        skip=`echo "$$results" | grep -c '^SKIP'`;                      \
        xfail=`echo "$$results" | grep -c '^XFAIL'`;                    \
        xpass=`echo "$$results" | grep -c '^XPASS'`;                    \
+       error=`echo "$$results" | grep -c '^ERROR'`;                    \
+## FIXME: for the moment, we count errors as failures, otherwise the code
+## that displays the testsuite summary will become too complicated.
+       fail=`expr $$fail + $$error`;                                   \
        failures=`expr $$fail + $$xpass`;                               \
        all=`expr $$all - $$skip`;                                      \
        if test "$$all" -eq 1; then tests=test; All=;                   \
diff --git a/lib/test-driver b/lib/test-driver
index 145bbea..aa39c5b 100755
--- a/lib/test-driver
+++ b/lib/test-driver
@@ -85,9 +85,10 @@ if test $color_tests = yes; then
   grn='[0;32m' # Green.
   lgn='[1;32m' # Light green.
   blu='[1;34m' # Blue.
+  mgn='[0;35m' # Magenta.
   std='[m'     # No color.
 else
-  red= grn= lgn= blu= std=
+  red= grn= lgn= blu= mgn= std=
 fi
 
 tmpfile=$logfile-t
@@ -109,7 +110,7 @@ case $estatus:$expect_failure in
   0:yes) col=$red; res=XPASS;;
   0:*)   col=$grn; res=PASS ;;
   77:*)  col=$blu; res=SKIP ;;
-  99:*)  col=$red; res=FAIL ;;
+  99:*)  col=$mgn; res=ERROR;;
   *:yes) col=$lgn; res=XFAIL;;
   *:*)   col=$red; res=FAIL ;;
 esac
diff --git a/tests/Makefile.am b/tests/Makefile.am
index 9c0605d..64299b8 100644
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -731,6 +731,8 @@ parallel-tests-log-override-recheck.test \
 parallel-tests-empty-testlogs.test \
 parallel-test-driver-install.test \
 parallel-tests-no-spurious-summary.test \
+parallel-tests-exit-statuses.test \
+parallel-tests-console-output.test \
 test-driver-end-test-results.test \
 test-driver-custom-no-extra-driver.test \
 test-driver-custom.test \
diff --git a/tests/Makefile.in b/tests/Makefile.in
index dc03179..5ef82da 100644
--- a/tests/Makefile.in
+++ b/tests/Makefile.in
@@ -991,6 +991,8 @@ parallel-tests-log-override-recheck.test \
 parallel-tests-empty-testlogs.test \
 parallel-test-driver-install.test \
 parallel-tests-no-spurious-summary.test \
+parallel-tests-exit-statuses.test \
+parallel-tests-console-output.test \
 test-driver-end-test-results.test \
 test-driver-custom-no-extra-driver.test \
 test-driver-custom.test \
@@ -1400,7 +1402,7 @@ $(TEST_SUITE_LOG): $(TEST_LOGS)
        nlinit=`echo 'nl="'; echo '"'`; eval "$$nlinit"; unset nlinit; \
        list='$(TEST_LOGS)'; \
        list2=`for f in $$list; do test ! -r $$f || echo $$f; done`; \
-       results1=`for f in $$list; do test -r $$f || echo FAIL; done`; \
+       results1=`for f in $$list; do test -r $$f || echo ERROR; done`; \
        results2=''; \
        exec 5<&0; \
        for f in $$list2; do \
@@ -1421,6 +1423,8 @@ $(TEST_SUITE_LOG): $(TEST_LOGS)
        skip=`echo "$$results" | grep -c '^SKIP'`;                      \
        xfail=`echo "$$results" | grep -c '^XFAIL'`;                    \
        xpass=`echo "$$results" | grep -c '^XPASS'`;                    \
+       error=`echo "$$results" | grep -c '^ERROR'`;                    \
+       fail=`expr $$fail + $$error`;                                   \
        failures=`expr $$fail + $$xpass`;                               \
        all=`expr $$all - $$skip`;                                      \
        if test "$$all" -eq 1; then tests=test; All=;                   \
diff --git a/tests/color.test b/tests/color.test
index ecd4f94..2524324 100755
--- a/tests/color.test
+++ b/tests/color.test
@@ -22,11 +22,14 @@
 TERM=ansi
 export TERM
 
-red='[0;31m'
-grn='[0;32m'
-lgn='[1;32m'
-blu='[1;34m'
-std='[m'
+esc=''
+# Escape `[' for grep, below.
+red="$esc\[0;31m"
+grn="$esc\[0;32m"
+lgn="$esc\[1;32m"
+blu="$esc\[1;34m"
+mgn="$esc\[0;35m"
+std="$esc\[m"
 
 # Check that grep can parse nonprinting characters.
 # BSD 'grep' works from a pipe, but not a seekable file.
@@ -43,7 +46,7 @@ END
 cat >Makefile.am <<'END'
 AUTOMAKE_OPTIONS = color-tests
 TESTS = $(check_SCRIPTS)
-check_SCRIPTS = pass fail skip xpass xfail
+check_SCRIPTS = pass fail skip xpass xfail error
 XFAIL_TESTS = xpass xfail
 END
 
@@ -62,39 +65,89 @@ cat >skip <<END
 exit 77
 END
 
+cat >error <<END
+#! /bin/sh
+exit 99
+END
+
 cp fail xfail
 cp pass xpass
-chmod +x pass fail skip xpass xfail
+chmod +x pass fail skip xpass xfail error
 
 $ACLOCAL
 $AUTOCONF
 $AUTOMAKE --add-missing
 
-./configure
-
 test_color ()
 {
   # Not a useless use of cat; see above comments about grep.
-  cat stdout | grep ": pass" | $FGREP "$grn"
-  cat stdout | grep ": fail" | $FGREP "$red"
-  cat stdout | grep ": xfail" | $FGREP "$lgn"
-  cat stdout | grep ": xpass" | $FGREP "$red"
-  cat stdout | grep ": skip" | $FGREP "$blu"
+  cat stdout | grep "^${grn}PASS${std}: .*pass"
+  cat stdout | grep "^${red}FAIL${std}: .*fail"
+  cat stdout | grep "^${blu}SKIP${std}: .*skip"
+  cat stdout | grep "^${lgn}XFAIL${std}: .*xfail"
+  cat stdout | grep "^${red}XPASS${std}: .*xpass"
+  # The old serial testsuite driver doesn't distinguish between failures
+  # and hard errors.
+  if test x"$parallel_tests" = x"yes"; then
+    cat stdout | grep "^${mgn}ERROR${std}: .*error"
+  else
+    cat stdout | grep "^${red}FAIL${std}: .*error"
+  fi
+  :
 }
 
 test_no_color ()
 {
-  # Not a useless use of cat; see above comments about grep.
-  cat stdout | grep ": pass" | $FGREP "$grn" && Exit 1
-  cat stdout | grep ": fail" | $FGREP "$red" && Exit 1
-  cat stdout | grep ": xfail" | $FGREP "$lgn" && Exit 1
-  cat stdout | grep ": xpass" | $FGREP "$red" && Exit 1
-  cat stdout | grep ": skip" | $FGREP "$blu" && Exit 1
-  :
+  # With make implementations that, like Solaris make, in case of errors
+  # print the whole failing recipe on standard output, we should content
+  # ourselves with a laxer check, to avoid false positives.
+  # Keep this in sync with lib/am/check.am:$(am__color_tests).
+  if $FGREP '= Xalways || test -t 1 ' stdout; then
+    # Extra verbose make, resort to laxer checks.
+    (
+      set +e # In case some grepped regex below isn't matched.
+      # Not a useless use of cat; see above comments about grep.
+      cat stdout | grep "PASS.*:"
+      cat stdout | grep "FAIL.*:"
+      cat stdout | grep "SKIP.*:"
+      cat stdout | grep "XFAIL.*:"
+      cat stdout | grep "XPASS.*:"
+      cat stdout | grep "ERROR.*:"
+      # To check that the testsuite summary is not unduly colorized.
+      cat stdout | grep '===='
+      cat stdout | grep 'test.*expected'
+      cat stdout | grep 'test.*not run'
+    ) | grep "$esc" && Exit 1
+    : For shells with broken 'set -e'
+  else
+    cat stdout | grep "$esc" && Exit 1
+    : For shells with broken 'set -e'
+  fi
 }
 
-AM_COLOR_TESTS=always $MAKE -e check >stdout && { cat stdout; Exit 1; }
-cat stdout
-test_color
+for vpath in false :; do
+
+  if $vpath; then
+    mkdir build
+    cd build
+    srcdir=..
+  else
+    srcdir=.
+  fi
+
+  $srcdir/configure
+
+  AM_COLOR_TESTS=always $MAKE -e check >stdout && { cat stdout; Exit 1; }
+  cat stdout
+  test_color
+
+  $MAKE -e check >stdout && { cat stdout; Exit 1; }
+  cat stdout
+  test_no_color
+
+  $MAKE distclean
+  cd $srcdir
+
+done
 
 :
diff --git a/tests/color2.test b/tests/color2.test
index 4f8862e..758ccd4 100755
--- a/tests/color2.test
+++ b/tests/color2.test
@@ -22,11 +22,14 @@
 TERM=ansi
 export TERM
 
-red='[0;31m'
-grn='[0;32m'
-lgn='[1;32m'
-blu='[1;34m'
-std='[m'
+esc=''
+# Escape `[' for grep, below.
+red="$esc\[0;31m"
+grn="$esc\[0;32m"
+lgn="$esc\[1;32m"
+blu="$esc\[1;34m"
+mgn="$esc\[0;35m"
+std="$esc\[m"
 
 # Check that grep can parse nonprinting characters.
 # BSD 'grep' works from a pipe, but not a seekable file.
@@ -56,7 +59,7 @@ END
 cat >Makefile.am <<'END'
 AUTOMAKE_OPTIONS = color-tests
 TESTS = $(check_SCRIPTS)
-check_SCRIPTS = pass fail skip xpass xfail
+check_SCRIPTS = pass fail skip xpass xfail error
 XFAIL_TESTS = xpass xfail
 END
 
@@ -75,34 +78,64 @@ cat >skip <<END
 exit 77
 END
 
+cat >error <<END
+#! /bin/sh
+exit 99
+END
+
 cp fail xfail
 cp pass xpass
-chmod +x pass fail skip xpass xfail
+chmod +x pass fail skip xpass xfail error
 
 $ACLOCAL
-$AUTOMAKE -a
 $AUTOCONF
-./configure
+$AUTOMAKE --add-missing
 
 test_color ()
 {
   # Not a useless use of cat; see above comments about grep.
-  cat stdout | grep ": pass" | $FGREP "$grn"
-  cat stdout | grep ": fail" | $FGREP "$red"
-  cat stdout | grep ": xfail" | $FGREP "$lgn"
-  cat stdout | grep ": xpass" | $FGREP "$red"
-  cat stdout | grep ": skip" | $FGREP "$blu"
+  cat stdout | grep "^${grn}PASS${std}: .*pass"
+  cat stdout | grep "^${red}FAIL${std}: .*fail"
+  cat stdout | grep "^${blu}SKIP${std}: .*skip"
+  cat stdout | grep "^${lgn}XFAIL${std}: .*xfail"
+  cat stdout | grep "^${red}XPASS${std}: .*xpass"
+  # The old serial testsuite driver doesn't distinguish between failures
+  # and hard errors.
+  if test x"$parallel_tests" = x"yes"; then
+    cat stdout | grep "^${mgn}ERROR${std}: .*error"
+  else
+    cat stdout | grep "^${red}FAIL${std}: .*error"
+  fi
+  :
 }
 
 test_no_color ()
 {
-  # Not a useless use of cat; see above comments about grep.
-  cat stdout | grep ": pass" | $FGREP "$grn" && Exit 1
-  cat stdout | grep ": fail" | $FGREP "$red" && Exit 1
-  cat stdout | grep ": xfail" | $FGREP "$lgn" && Exit 1
-  cat stdout | grep ": xpass" | $FGREP "$red" && Exit 1
-  cat stdout | grep ": skip" | $FGREP "$blu" && Exit 1
-  :
+  # With make implementations that, like Solaris make, in case of errors
+  # print the whole failing recipe on standard output, we should content
+  # ourselves with a laxer check, to avoid false positives.
+  # Keep this in sync with lib/am/check.am:$(am__color_tests).
+  if $FGREP '= Xalways || test -t 1 ' stdout; then
+    # Extra verbose make, resort to laxer checks.
+    (
+      set +e # In case some grepped regex below isn't matched.
+      # Not a useless use of cat; see above comments about grep.
+      cat stdout | grep "PASS.*:"
+      cat stdout | grep "FAIL.*:"
+      cat stdout | grep "SKIP.*:"
+      cat stdout | grep "XFAIL.*:"
+      cat stdout | grep "XPASS.*:"
+      cat stdout | grep "ERROR.*:"
+      # To check that the testsuite summary is not unduly colorized.
+      cat stdout | grep '===='
+      cat stdout | grep 'test.*expected'
+      cat stdout | grep 'test.*not run'
+    ) | grep "$esc" && Exit 1
+    : For shells with broken 'set -e'
+  else
+    cat stdout | grep "$esc" && Exit 1
+    : For shells with broken 'set -e'
+  fi
 }
 
 cat >expect-make <<'END'
@@ -111,14 +144,31 @@ eval spawn $env(MAKE) -e check
 expect eof
 END
 
-MAKE=$MAKE expect -f expect-make >stdout \
-  || { cat stdout; Exit 1; }
-cat stdout
-test_color
+for vpath in false :; do
+
+  if $vpath; then
+    mkdir build
+    cd build
+    srcdir=..
+  else
+    srcdir=.
+  fi
+
+  $srcdir/configure
+
+  MAKE=$MAKE expect -f $srcdir/expect-make >stdout \
+    || { cat stdout; Exit 1; }
+  cat stdout
+  test_color
+
+  AM_COLOR_TESTS=no MAKE=$MAKE expect -f $srcdir/expect-make >stdout \
+    || { cat stdout; Exit 1; }
+  cat stdout
+  test_no_color
+
+  $MAKE distclean
+  cd $srcdir
 
-AM_COLOR_TESTS=no MAKE=$MAKE expect -f expect-make >stdout \
-  || { cat stdout; Exit 1; }
-cat stdout
-test_no_color
+done
 
 :
diff --git a/tests/parallel-tests-console-output.test 
b/tests/parallel-tests-console-output.test
new file mode 100755
index 0000000..8122bfd
--- /dev/null
+++ b/tests/parallel-tests-console-output.test
@@ -0,0 +1,104 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+# parallel-tests: some checks on console output about testsuite
+# progress.
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cat >> configure.in << 'END'
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+XFAIL_TESTS = sub/xpass.test xfail.test error.test
+TESTS = $(XFAIL_TESTS) fail.test pass.test a/b/skip.test sub/error2.test
+pass.log: fail.log
+error.log: pass.log
+sub/xpass.log: error.log
+sub/error2.log: xfail.log
+a/b/skip.log: sub/error2.log
+END
+
+cat > exp <<'END'
+FAIL: fail.test
+PASS: pass.test
+ERROR: error.test
+XPASS: sub/xpass.test
+XFAIL: xfail.test
+ERROR: sub/error2.test
+SKIP: a/b/skip.test
+END
+
+mkdir sub a a/b
+
+cat > pass.test << 'END'
+#!/bin/sh
+exit 0
+END
+cp pass.test sub/xpass.test
+
+cat > fail.test << 'END'
+#!/bin/sh
+exit 1
+END
+
+cat > xfail.test << 'END'
+#!/bin/sh
+# The sleep should ensure expected execution order of tests
+# even when make is run in parallel mode.
+# FIXME: quotes below required by maintainer-check.
+sleep '10'
+exit 1
+END
+
+cat > error.test << 'END'
+#!/bin/sh
+exit 99
+END
+cp error.test sub/error2.test
+
+cat > a/b/skip.test << 'END'
+#!/bin/sh
+exit 77
+END
+
+chmod a+x pass.test fail.test xfail.test sub/xpass.test \
+          a/b/skip.test error.test sub/error2.test
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE -a
+
+for vpath in : false; do
+  if $vpath; then
+    mkdir build
+    cd build
+    srcdir=..
+  else
+    srcdir=.
+  fi
+  $srcdir/configure
+  $MAKE check >stdout && { cat stdout; Exit 1; }
+  cat stdout
+  LC_ALL=C grep '^[A-Z][A-Z]*:' stdout > got
+  cat got
+  diff $srcdir/exp got
+  cd $srcdir
+done
+
+:
diff --git a/tests/parallel-tests-exit-statuses.test 
b/tests/parallel-tests-exit-statuses.test
new file mode 100755
index 0000000..531526a
--- /dev/null
+++ b/tests/parallel-tests-exit-statuses.test
@@ -0,0 +1,108 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+# Check parallel-tests features: normal and special exit statuses
+# in the test scripts.
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cat >> configure.in << 'END'
+AC_OUTPUT
+END
+
+# $failure_statuses should be defined to the list of all integers between
+# 1 and 255 (inclusive), excluded 77 and 99.
+# Let's use `seq' if available, it's faster than the loop.
+failure_statuses=`seq 1 255 \
+  || { i=1; while test $i -le 255; do echo $i; i=\`expr $i + 1\`; done; }`
+failure_statuses=`
+  for i in $failure_statuses; do
+    test $i -eq 77 || test $i -eq 99 || echo $i
+  done | tr "$nl" ' '`
+# For debugging.
+echo "failure_statuses: $failure_statuses"
+# Sanity check.
+test `for st in $failure_statuses; do echo $st; done | wc -l` -eq 253 \
+  || fatal_ "initializing list of exit statuses for simple failures"
+
+cat > Makefile.am <<END
+LOG_COMPILER = ./do-exit
+fail_tests = $failure_statuses
+TESTS = 0 77 99 $failure_statuses
+\$(TESTS):
+END
+
+cat > do-exit <<'END'
+#!/bin/sh
+echo "$0: $1"
+case $1 in
+  [0-9]|[0-9][0-9]|[0-9][0-9][0-9]) st=$1;;
+  */[0-9]|*/[0-9][0-9]|*/[0-9][0-9][0-9]) st=`echo x"$1" | sed 's|.*/||'`;;
+  *) st=99;;
+esac
+exit $st
+END
+chmod a+x do-exit
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE -a
+
+{
+  echo PASS: 0
+  echo SKIP: 77
+  echo ERROR: 99
+  for st in $failure_statuses; do
+    echo "FAIL: $st"
+  done
+} | LC_ALL=C sort > exp-fail
+
+sed 's/^FAIL:/XFAIL:/' exp-fail | LC_ALL=C sort > exp-xfail-1
+sed '/^ERROR:/d' exp-xfail-1 > exp-xfail-2
+
+sort exp-fail
+sort exp-xfail-1
+sort exp-xfail-2
+
+./configure
+
+st=1
+$MAKE check >stdout && st=0
+cat stdout
+cat test-suite.log
+test $st -gt 0 || Exit 1
+LC_ALL=C grep '^[A-Z][A-Z]*:' stdout | LC_ALL=C sort > got-fail
+diff exp-fail got-fail
+
+st=1
+XFAIL_TESTS="$failure_statuses 99" $MAKE -e check >stdout && st=0
+cat stdout
+cat test-suite.log
+test $st -gt 0 || Exit 1
+LC_ALL=C grep '^[A-Z][A-Z]*:' stdout | LC_ALL=C sort > got-xfail-1
+diff exp-xfail-1 got-xfail-1
+
+st=0
+XFAIL_TESTS="$failure_statuses" TESTS="0 77 $failure_statuses" \
+  $MAKE -e check >stdout || st=$?
+cat stdout
+cat test-suite.log
+test $st -eq 0 || Exit 1
+LC_ALL=C grep '^[A-Z][A-Z]*:' stdout | LC_ALL=C sort > got-xfail-2
+diff exp-xfail-2 got-xfail-2
+
+:
diff --git a/tests/parallel-tests-harderror.test 
b/tests/parallel-tests-harderror.test
index 8680e29..f30fd52 100755
--- a/tests/parallel-tests-harderror.test
+++ b/tests/parallel-tests-harderror.test
@@ -69,13 +69,13 @@ DISABLE_HARD_ERRORS=x $MAKE check
 
 $MAKE check DISABLE_HARD_ERRORS='' && Exit 1
 cat test-suite.log
-grep '^FAIL: foo\.test .*exit.*99' test-suite.log
+grep '^ERROR: foo\.test .*exit.*99' test-suite.log
 
 cd sub
 # The `-e' is wanted here.
 DISABLE_HARD_ERRORS='' $MAKE -e check && Exit 1
 cat test-suite.log
-grep '^FAIL: bar\.test .*exit.*99' test-suite.log
+grep '^ERROR: bar\.test .*exit.*99' test-suite.log
 cd ..
 
 # Check the distributions.
@@ -90,7 +90,7 @@ $AUTOMAKE Makefile
 ./config.status Makefile
 VERBOSE=yes $MAKE check && Exit 1
 grep '^FAIL' test-suite.log && Exit 1
-grep '^FAIL: bar\.test .*exit.*99' sub/test-suite.log
+grep '^ERROR: bar\.test .*exit.*99' sub/test-suite.log
 
 echo 'DISABLE_HARD_ERRORS = zardoz' >> sub/Makefile
 VERBOSE=yes $MAKE check
diff --git a/tests/parallel-tests-no-spurious-summary.test 
b/tests/parallel-tests-no-spurious-summary.test
index 43c54a5..8b9547a 100755
--- a/tests/parallel-tests-no-spurious-summary.test
+++ b/tests/parallel-tests-no-spurious-summary.test
@@ -32,10 +32,12 @@ cat > foo.test <<'END'
 #! /bin/sh
 echo :test-result:XFAIL
 echo :test-result: SKIP
+echo :test-result:ERROR
 exit 0
 END
 cat > bar.test <<'END'
 #! /bin/sh
+echo :test-result: ERROR
 echo :test-result:FAIL
 echo :test-result: XPASS
 exit 0
diff --git a/tests/parallel-tests.test b/tests/parallel-tests.test
index f046c3f..95f78e7 100755
--- a/tests/parallel-tests.test
+++ b/tests/parallel-tests.test
@@ -68,10 +68,13 @@ $AUTOMAKE -a
 
 $MAKE check >stdout && { cat stdout; Exit 1; }
 cat stdout
-# There should be two errors: bar.test is a hard error.
-test `grep -c '^FAIL' stdout` -eq 2
+# There should be one failure and one hard error.
+test `grep -c '^FAIL:' stdout` -eq 1
+test `grep -c '^ERROR:' stdout` -eq 1
 test -f mylog.log
-test `grep -c '^FAIL' mylog.log` -eq 2
+cat mylog.log
+test `grep -c '^FAIL:' mylog.log` -eq 1
+test `grep -c '^ERROR:' mylog.log` -eq 1
 test -f baz.log
 test -f bar.log
 test -f foo.log
diff --git a/tests/test-driver-custom-multitest-recheck.test 
b/tests/test-driver-custom-multitest-recheck.test
index e0f010f..c178fa0 100755
--- a/tests/test-driver-custom-multitest-recheck.test
+++ b/tests/test-driver-custom-multitest-recheck.test
@@ -49,7 +49,7 @@ echo PASS:
 if test -f b.ok; then
   echo PASS:
 else
-  echo FAIL:
+  echo ERROR:
 fi
 : > b.run
 END
@@ -97,14 +97,15 @@ do_recheck ()
 
 do_count ()
 {
-  pass=ERR fail=ERR xpass=ERR xfail=ERR skip=ERR
+  pass=ERR fail=ERR xpass=ERR xfail=ERR skip=ERR error=ERR
   eval "$@"
-  $EGREP '(PASS|FAIL|XPASS|XFAIL|SKIP)' stdout || : # For debugging.
+  $EGREP '(PASS|FAIL|XPASS|XFAIL|SKIP|ERROR)' stdout || : # For debugging.
   test `grep -c '^PASS:' stdout` -eq $pass
   test `grep -c '^FAIL:' stdout` -eq $fail
   test `grep -c '^XPASS:' stdout` -eq $xpass
   test `grep -c '^XFAIL:' stdout` -eq $xfail
   test `grep -c '^SKIP:' stdout` -eq $skip
+  test `grep -c '^ERROR:' stdout` -eq $error
 }
 
 for vpath in : false; do
@@ -129,7 +130,7 @@ for vpath in : false; do
   test ! -r c.log
   test ! -r d.run
   test ! -r d.log
-  do_count pass=0 fail=0 xpass=0 xfail=0 skip=0
+  do_count pass=0 fail=0 xpass=0 xfail=0 skip=0 error=0
 
   : Run the tests for the first time.
   $MAKE check >stdout && { cat stdout; Exit 1; }
@@ -140,7 +141,7 @@ for vpath in : false; do
   test -f b.run
   test -f c.run
   test -f d.run
-  do_count pass=3 fail=3 xpass=1 xfail=1 skip=1
+  do_count pass=3 fail=2 xpass=1 xfail=1 skip=1 error=1
 
   : Let us make b.test pass.
   echo OK > b.ok
@@ -151,7 +152,7 @@ for vpath in : false; do
   test -f b.run
   test -f c.run
   test -f d.run
-  do_count pass=2 fail=2 xpass=1 xfail=1 skip=1
+  do_count pass=2 fail=2 xpass=1 xfail=1 skip=1 error=0
 
   : Let us make the first part of c.test pass.
   echo OK > c.pass
@@ -160,7 +161,7 @@ for vpath in : false; do
   test ! -r b.run
   test -f c.run
   test -f d.run
-  do_count pass=1 fail=1 xpass=1 xfail=1 skip=1
+  do_count pass=1 fail=1 xpass=1 xfail=1 skip=1 error=0
 
   : Let us make also the second part of c.test pass.
   echo KO > c.xfail
@@ -169,7 +170,7 @@ for vpath in : false; do
   test ! -r b.run
   test -f c.run
   test -f d.run
-  do_count pass=1 fail=1 xpass=0 xfail=2 skip=1
+  do_count pass=1 fail=1 xpass=0 xfail=2 skip=1 error=0
 
   : Nothing changed, so only d.test should be run.
   for i in 1 2; do
@@ -178,7 +179,7 @@ for vpath in : false; do
     test ! -r b.run
     test ! -r c.run
     test -f d.run
-    do_count pass=0 fail=1 xpass=0 xfail=0 skip=1
+    do_count pass=0 fail=1 xpass=0 xfail=0 skip=1 error=0
   done
 
   : Let us make d.test run more testcases, and experience _more_ failures.
@@ -190,14 +191,16 @@ for vpath in : false; do
     echo XPASS: xp
     echo FAIL: f 3
     echo FAIL: f 4
+    echo ERROR: e 1
     echo PASS: p 2
+    echo ERROR: e 2
 END
   do_recheck --fail
   test ! -r a.run
   test ! -r b.run
   test ! -r c.run
   test -f d.run
-  do_count pass=2 fail=4 xpass=1 xfail=0 skip=2
+  do_count pass=2 fail=4 xpass=1 xfail=0 skip=2 error=2
 
   : Let us finally make d.test pass.
   echo : > d.extra
@@ -206,7 +209,7 @@ END
   test ! -r b.run
   test ! -r c.run
   test -f d.run
-  do_count pass=0 fail=0 xpass=0 xfail=0 skip=1
+  do_count pass=0 fail=0 xpass=0 xfail=0 skip=1 error=0
 
   : All tests have been successful or skipped, nothing should be re-run.
   do_recheck --pass
@@ -214,7 +217,7 @@ END
   test ! -r b.run
   test ! -r c.run
   test ! -r d.run
-  do_count pass=0 fail=0 xpass=0 xfail=0 skip=0
+  do_count pass=0 fail=0 xpass=0 xfail=0 skip=0 error=0
 
   cd $srcdir
 
diff --git a/tests/test-driver-custom-multitest-recheck2.test 
b/tests/test-driver-custom-multitest-recheck2.test
index a18bdd8..36cb302 100755
--- a/tests/test-driver-custom-multitest-recheck2.test
+++ b/tests/test-driver-custom-multitest-recheck2.test
@@ -56,7 +56,13 @@ END
 
 cat > c.test << 'END'
 #! /bin/sh
-echo XPASS: xp
+if test -f c.err; then
+  echo ERROR: xxx
+elif test -f c.ok; then
+  echo PASS: ok
+else
+  echo XPASS: xp
+fi
 : > c.run
 END
 
@@ -68,14 +74,15 @@ $AUTOMAKE
 
 do_count ()
 {
-  pass=ERR fail=ERR xpass=ERR xfail=ERR skip=ERR
+  pass=ERR fail=ERR xpass=ERR xfail=ERR skip=ERR error=ERR
   eval "$@"
-  $EGREP '(PASS|FAIL|XPASS|XFAIL|SKIP)' stdout || : # For debugging.
+  $EGREP '(PASS|FAIL|XPASS|XFAIL|SKIP|ERROR)' stdout || : # For debugging.
   test `grep -c '^PASS:' stdout` -eq $pass
   test `grep -c '^FAIL:' stdout` -eq $fail
   test `grep -c '^XPASS:' stdout` -eq $xpass
   test `grep -c '^XFAIL:' stdout` -eq $xfail
   test `grep -c '^SKIP:' stdout` -eq $skip
+  test `grep -c '^ERROR:' stdout` -eq $error
 }
 
 for vpath in : false; do
@@ -96,7 +103,7 @@ for vpath in : false; do
   test -f a.run
   test -f b.run
   test -f c.run
-  do_count pass=2 fail=1 xpass=1 xfail=0 skip=1
+  do_count pass=2 fail=1 xpass=1 xfail=0 skip=1 error=0
 
   rm -f *.run
 
@@ -104,7 +111,7 @@ for vpath in : false; do
   for var in TESTS TEST_LOGS; do
     env "$var=" $MAKE -e recheck >stdout || { cat stdout; Exit 1; }
     cat stdout
-    do_count pass=0 fail=0 xpass=0 xfail=0 skip=0
+    do_count pass=0 fail=0 xpass=0 xfail=0 skip=0 error=0
     test ! -r a.run
     test ! -r b.run
     test ! -r c.run
@@ -115,7 +122,7 @@ for vpath in : false; do
   env TESTS=a.test $MAKE -e recheck >stdout \
     || { cat stdout; Exit 1; }
   cat stdout
-  do_count pass=0 fail=0 xpass=0 xfail=0 skip=0
+  do_count pass=0 fail=0 xpass=0 xfail=0 skip=0 error=0
   test ! -r a.run
   test ! -r b.run
   test ! -r c.run
@@ -128,7 +135,7 @@ for vpath in : false; do
   test ! -r a.run
   test -f b.run
   test ! -r c.run
-  do_count pass=0 fail=0 xpass=0 xfail=1 skip=1
+  do_count pass=0 fail=0 xpass=0 xfail=1 skip=1 error=0
 
   rm -f *.run
 
@@ -136,35 +143,54 @@ for vpath in : false; do
   TEST_LOGS=b.log $MAKE -e recheck >stdout \
     || { cat stdout; Exit 1; }
   cat stdout
-  do_count pass=0 fail=0 xpass=0 xfail=0 skip=0
+  do_count pass=0 fail=0 xpass=0 xfail=0 skip=0 error=0
   test ! -r a.run
   test ! -r b.run
   test ! -r c.run
   TESTS='a.test b.test' $MAKE -e recheck >stdout \
     || { cat stdout; Exit 1; }
   cat stdout
-  do_count pass=0 fail=0 xpass=0 xfail=0 skip=0
+  do_count pass=0 fail=0 xpass=0 xfail=0 skip=0 error=0
   test ! -r a.run
   test ! -r b.run
   test ! -r c.run
 
-  # An XPASS should count a failure.
+  : No need to re-run a.test anymore, but c.test should be rerun,
+  : as it contained an XPASS.  And this time, make it fail with
+  : an hard error.
+  : > c.err
   env TEST_LOGS='a.log c.log' $MAKE -e recheck >stdout \
     && { cat stdout; Exit 1; }
   cat stdout
-  do_count pass=0 fail=0 xpass=1 xfail=0 skip=0
+  do_count pass=0 fail=0 xpass=0 xfail=0 skip=0 error=1
   test ! -r a.run
   test ! -r b.run
   test -f c.run
-  rm -f *.run
-  env TESTS='c.test b.test' $MAKE -e recheck >stdout \
-    && { cat stdout; Exit 1; }
+
+  rm -f *.run *.err
+
+  : c.test contained and hard error the last time, so it should be re-run.
+  : This time, make it pass
+  : > c.ok
+  env TESTS='c.test a.test' $MAKE -e recheck >stdout \
+    || { cat stdout; Exit 1; }
   cat stdout
-  do_count pass=0 fail=0 xpass=1 xfail=0 skip=0
+  do_count pass=1 fail=0 xpass=0 xfail=0 skip=0 error=0
   test ! -r a.run
   test ! -r b.run
   test -f c.run
 
+  rm -f *.run *.err *.ok
+
+  : Nothing should be rerun anymore, as all tests have been eventually
+  : succesful.
+  $MAKE recheck >stdout || { cat stdout; Exit 1; }
+  cat stdout
+  do_count pass=0 fail=0 xpass=0 xfail=0 skip=0 error=0
+  test ! -r a.run
+  test ! -r b.run
+  test ! -r c.run
+
   cd $srcdir
 
 done
diff --git a/tests/test-driver-custom-multitest.test 
b/tests/test-driver-custom-multitest.test
index a1c1010..2366b5b 100755
--- a/tests/test-driver-custom-multitest.test
+++ b/tests/test-driver-custom-multitest.test
@@ -40,7 +40,7 @@ TESTS = \
   pass-fail.t \
   pass4-skip.t \
   pass3-skip2-xfail.t \
-  pass-xpass-fail-xfail-skip.t
+  pass-xpass-fail-xfail-skip-error.t
 END
 
 expected_pass=10
@@ -48,6 +48,7 @@ expected_fail=5
 expected_skip=4
 expected_xfail=2
 expected_xpass=1
+expected_error=1
 
 cat > pass.t << 'END'
 echo %% pass %%
@@ -99,13 +100,14 @@ mv || echo "XFAIL: mv cannot read users' mind yet"
 exit 127
 END
 
-cat > pass-xpass-fail-xfail-skip.t << 'END'
+cat > pass-xpass-fail-xfail-skip-error.t << 'END'
 echo PASS:
 echo FAIL:
 echo XFAIL:
 echo XPASS:
 echo SKIP:
-echo %% pass-xpass-fail-xfail-skip %%
+echo ERROR:
+echo %% pass-xpass-fail-xfail-skip-error %%
 END
 
 chmod a+x *.t
@@ -139,25 +141,28 @@ for vpath in : false; do
   cat pass-fail.log
   cat pass4-skip.log
   cat pass3-skip2-xfail.log
-  cat pass-xpass-fail-xfail-skip.log
+  cat pass-xpass-fail-xfail-skip-error.log
   # For debugging.
-  $EGREP '(PASS|FAIL|XPASS|XFAIL|SKIP)' stdout
+  $EGREP '(PASS|FAIL|XPASS|XFAIL|SKIP|ERROR)' stdout
 
   test `grep -c '^PASS:' stdout` -eq $expected_pass
   test `grep -c '^FAIL:' stdout` -eq $expected_fail
   test `grep -c '^XPASS:' stdout` -eq $expected_xpass
   test `grep -c '^XFAIL:' stdout` -eq $expected_xfail
   test `grep -c '^SKIP:' stdout` -eq $expected_skip
+  test `grep -c '^ERROR:' stdout` -eq $expected_error
 
-  grep  '^PASS: pass-xpass-fail-xfail-skip.t\, testcase 1' stdout
-  grep  '^FAIL: pass-xpass-fail-xfail-skip\.t, testcase 2' stdout
-  grep '^XFAIL: pass-xpass-fail-xfail-skip\.t, testcase 3' stdout
-  grep '^XPASS: pass-xpass-fail-xfail-skip\.t, testcase 4' stdout
-  grep  '^SKIP: pass-xpass-fail-xfail-skip\.t, testcase 5' stdout
+  tst=pass-xpass-fail-xfail-skip-error
+  grep  "^PASS: $tst\.t, testcase 1" stdout
+  grep  "^FAIL: $tst\.t, testcase 2" stdout
+  grep "^XFAIL: $tst\.t, testcase 3" stdout
+  grep "^XPASS: $tst\.t, testcase 4" stdout
+  grep  "^SKIP: $tst\.t, testcase 5" stdout
+  grep "^ERROR: $tst\.t, testcase 6" stdout
 
   # Check testsuite summary printed on console.
   sed -e 's/[()]/ /g' -e 's/^/ /' stdout > t
-  grep ' 6 of 18 ' t
+  grep ' 7 of 19 ' t
   grep ' 1 unexpected pass' t
   grep ' 4 test.* not run' t
 
@@ -170,7 +175,7 @@ for vpath in : false; do
   grep '%% fail %%' test-suite.log
   grep '%% fail2 %%' test-suite.log
   grep '%% pass-fail %%' test-suite.log
-  grep '%% pass-xpass-fail-xfail-skip %%' test-suite.log
+  grep '%% pass-xpass-fail-xfail-skip-error %%' test-suite.log
   test `grep -c '%% ' test-suite.log` -eq 4
 
   TESTS='pass.t pass3-skip2-xfail.t' $MAKE -e check >stdout \
@@ -178,11 +183,11 @@ for vpath in : false; do
   cat test-suite.log
   cat stdout
   # For debugging.
-  $EGREP '(PASS|FAIL|XPASS|XFAIL|SKIP)' stdout
+  $EGREP '(PASS|FAIL|XPASS|XFAIL|SKIP|ERROR)' stdout
   test `grep -c '^PASS:' stdout` -eq 4
   test `grep -c '^SKIP:' stdout` -eq 2
   test `grep -c '^XFAIL:' stdout` -eq 1
-  $EGREP '^(FAIL|XPASS)' stdout && Exit 1
+  $EGREP '^(FAIL|XPASS|ERROR)' stdout && Exit 1
 
   cd $srcdir
 
diff --git a/tests/test-driver-custom-no-html.test 
b/tests/test-driver-custom-no-html.test
index 9aceae6..683659b 100755
--- a/tests/test-driver-custom-no-html.test
+++ b/tests/test-driver-custom-no-html.test
@@ -62,6 +62,6 @@ $FGREP 'dummy title' test-suite.log
 # Sanity check: trying to produce HTML output should fail.
 $MAKE check-html >output 2>&1 && { cat output; Exit 1; }
 cat output
-$EGREP 'SEVERE|ERROR' output
+$FGREP SEVERE output
 
 :
diff --git a/tests/test-driver-end-test-results.test 
b/tests/test-driver-end-test-results.test
index 97c1f29..03af51b 100755
--- a/tests/test-driver-end-test-results.test
+++ b/tests/test-driver-end-test-results.test
@@ -57,6 +57,7 @@ PASS: b.test
 :test-result:PASS
 :test-result:END
 :test-result:FAIL
+:test-result:ERROR
 END
 
 cat > c.test <<END
diff --git a/tests/test-driver-global-log.test 
b/tests/test-driver-global-log.test
index cdaaff6..35b6fe1 100755
--- a/tests/test-driver-global-log.test
+++ b/tests/test-driver-global-log.test
@@ -44,12 +44,13 @@ END
 echo SKIP: skip.test > skip.test
 echo FAIL: fail.test > fail.test
 echo XPASS: xpass.test > xpass.test
+echo ERROR: error.test > error.test
 echo :test-result: PASS > fake-pass.test
 echo "$tab $tab$tab" > empty.test
 
 cat > Makefile.am << 'END'
 TEST_LOG_DRIVER = ./dummy-driver
-TESTS = pass.test skip.test fail.test xfail.test xpass.test \
+TESTS = pass.test skip.test fail.test xfail.test xpass.test error.test \
         fake-pass.test empty.test
 END
 
@@ -82,6 +83,7 @@ cat test-suite.log
 grep '^SKIP: skip\.test$' test-suite.log
 grep '^FAIL: fail.test$' test-suite.log
 grep '^XPASS: xpass.test$' test-suite.log
+grep '^ERROR: error.test$' test-suite.log
 grep '^:test-result: PASS$' test-suite.log
 grep "^$tab $tab$tab$" test-suite.log
 $EGREP 'not seen' test-suite.log && Exit 1
diff --git a/tests/test-driver-recheck.test b/tests/test-driver-recheck.test
index 2ae75b9..f9482b1 100755
--- a/tests/test-driver-recheck.test
+++ b/tests/test-driver-recheck.test
@@ -54,7 +54,7 @@ END
   rechecked="$rechecked BAD-$R-1 BAD-$R-2 BAD-$R-3 BAD-$R-4"
 done
 
-for R in FAIL XPASS UNKNOWN; do
+for R in FAIL XPASS ERROR UNKNOWN; do
   echo $R: > $R-1
   echo $R:foo > $R-2
   echo $R: bar baz > $R-3
diff --git a/tests/trivial-test-driver b/tests/trivial-test-driver
index f06eb1c..113e158 100644
--- a/tests/trivial-test-driver
+++ b/tests/trivial-test-driver
@@ -20,9 +20,10 @@
 # test results per test script.
 #
 # The exit status of the wrapped script is ignored.  Lines in its stdout
-# and stderr beginning with `PASS', `FAIL', `XFAIL', `XPASS' or `SKIP'
-# count as a test case result with the obviously-corresponding outcome.
-# Every other line is ignored for what concerns the testsuite outcome.
+# and stderr beginning with `PASS', `FAIL', `XFAIL', `XPASS', `SKIP' or
+# `ERROR' count as a test case result with the obviously-corresponding
+# outcome.  Every other line is ignored for what concerns the testsuite
+# outcome.
 #
 # This script is used at least by the `driver-custom-multitest*.test'
 # tests.
@@ -61,10 +62,10 @@ tmp_res=$log_file-res.tmp
   : > $tmp_res
   while read line; do
     case $line in
-      PASS:*|FAIL:*|XPASS:*|XFAIL:*|SKIP:*)
+      PASS:*|FAIL:*|XPASS:*|XFAIL:*|SKIP:*|ERROR:*)
         i=`expr $i + 1`
         result=`LC_ALL=C expr "$line" : '\([A-Z]*\):.*'`
-        case $result in FAIL|XPASS) st=1;; esac
+        case $result in FAIL|XPASS|ERROR) st=1;; esac
         # Output testcase result to console.
         echo "$result: $test_name, testcase $i"
         # Register testcase outcome for the log file.
-- 
1.7.2.3
[Prev in Thread]
Current Thread
[Next in Thread]
[GSoC] Some patches for testsuite harness improvements and TAP support introduction, Stefano Lattarini, 2011/07/14
- [PATCH 1/5] {test-protocols} parallel-tests: make parsing of test results safer, Stefano Lattarini, 2011/07/14
  - Re: [PATCH 1/5] {test-protocols} parallel-tests: make parsing of test results safer, Ralf Wildenhues, 2011/07/18
    - Re: [PATCH 1/5] {test-protocols} parallel-tests: make parsing of test results safer, Stefano Lattarini, 2011/07/19
    - Re: [PATCH 1/5] {test-protocols} parallel-tests: make parsing of test results safer, Ralf Wildenhues, 2011/07/19
    - Re: [PATCH 1/5] {test-protocols} parallel-tests: make parsing of test results safer, Stefano Lattarini, 2011/07/20
    - Re: [PATCH 1/5] {test-protocols} parallel-tests: make parsing of test results safer, Ralf Wildenhues, 2011/07/20
    - Re: [PATCH 1/5] {test-protocols} parallel-tests: make parsing of test results safer, Stefano Lattarini, 2011/07/21
    - [PATCH 1/2] test harness: allow more metadata in log files, Stefano Lattarini, 2011/07/29
    - Re: [PATCH 1/2] test harness: allow more metadata in log files, Stefano Lattarini, 2011/07/31
- [PATCH 2/5] {test-protocols} parallel-tests: new recognized test result 'ERROR', Stefano Lattarini <=
  - Re: [PATCH 2/5] {test-protocols} parallel-tests: new recognized test result 'ERROR', Ralf Wildenhues, 2011/07/18
    - Re: [PATCH 2/5] {test-protocols} parallel-tests: new recognized test result 'ERROR', Stefano Lattarini, 2011/07/19
    - Re: [PATCH 2/5] {test-protocols} parallel-tests: new recognized test result 'ERROR', Ralf Wildenhues, 2011/07/19
    - [PATCH] {test-protocols} docs: rework and extend documentation on testsuites support, Stefano Lattarini, 2011/07/22
    - Re: [PATCH] {test-protocols} docs: rework and extend documentation on testsuites support, Stefano Lattarini, 2011/07/26
- [PATCH 3/5] {test-protocols} parallel-tests: simplify testsuite summary, Stefano Lattarini, 2011/07/14
- [PATCH 4/5] {test-protocols} tests defs: new auxiliary function 'count_test_results', Stefano Lattarini, 2011/07/14
  - Re: [PATCH 4/5] {test-protocols} tests defs: new auxiliary function 'count_test_results', Ralf Wildenhues, 2011/07/18
- [PATCH 5/5] {test-protocols} tap: add experimental TAP-aware driver, Stefano Lattarini, 2011/07/14
  - Re: [PATCH 5/5] {test-protocols} tap: add experimental TAP-aware driver, Stefano Lattarini, 2011/07/18
Prev by Date: [PATCH 1/5] {test-protocols} parallel-tests: make parsing of test results safer
Next by Date: [PATCH 3/5] {test-protocols} parallel-tests: simplify testsuite summary
Previous by thread: Re: [PATCH 1/2] test harness: allow more metadata in log files
Next by thread: Re: [PATCH 2/5] {test-protocols} parallel-tests: new recognized test result 'ERROR'
Index(es):
- Date
- Thread