[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [PATCH] {GSoC} parallel-tests: simplify testsuite summary
|
From: |
Stefano Lattarini |
|
Subject: |
Re: [PATCH] {GSoC} parallel-tests: simplify testsuite summary |
|
Date: |
Wed, 6 Jul 2011 14:49:15 +0200 |
|
User-agent: |
KMail/1.13.3 (Linux/2.6.30-2-686; KDE/4.4.4; i686; ; ) |
On Wednesday 06 July 2011, Ralf Wildenhues wrote:
> Hi Stefano,
>
> * Stefano Lattarini wrote on Sun, Jul 03, 2011 at 03:31:22PM CEST:
> > Prefer a more deterministic, "tabular" format for the testsuite
> > summary, always listing the numbers of passed, failed, xfailed,
> > xpassed, skipped and errored tests, even when these numbers are
> > zero. This simplify the logic of testsuite summary creation,
> > makes it more easily machine-parseable, and will probably allow
> > for easier addition of new kinds of test results in the future.
> >
> > Applied to the temporary branch 'GSoC/experimental/test-results-work'.
>
> This looks a bit verbose to me, also IIUC it introduces yet another
> format that is different from our previous summary and different from
> the summaries of any other testsuite environments. Any chance we can
> lessen the NIH?
>
Well, I'm surely open to revisit the issue later if we find an existing
format that offers all the information my new format does, is as easily
parseable, and does not suffer of the NIH syndrome :-). But the older
format has to go IMHO, for two orthogonal and good reasons:
- the logic necessary to create a summary that obeys that format is
complex, unclear, and not at all extensible; and
- it's pretty hard to verify the correctness of the resulting summary
by means of grepping checks.
> I've rejected a reformatting of the results before (see the list
> archives for details). I'm not totally opposed, if there is a real
> advantage /for the user/,
>
Personally I can say that, as a user, I find the new format definitely
clearer and more pleasant (at least when 'color-tests' is in use); and
this independently from the considerations above (that were admittedtly
done with the developer/tester hat on).
> and if things are changing a lot anyway,
> but we should definitely change into something that we think can be
> stable for several years.
>
The new format is much more easily extensible IMO, so it's easier for
it to remain stabler. Of course, there are some tweaking that might be
done before the next release -- for example, the field "tests" (which
suggests "number of tests that have been run") would better be labelled
as "total" (which suggests "total number of results that we've obtained"),
as also skipped tests and hard errors are counted. But this is minor
stuff, and easy to fix IMHO (before the release!).
... and now that I've thought about it, I'll make the change right away
of course ;-) See attached squash-in.
> This is end-user API, a lot of stuff that's
> not even using Automake (but only running a testsuite of a package that
> uses Automake) can be relying on this. IOW, things like GSRC, the
> autobuild parser, or whatnot else (that we don't have any control over)
> that greps our output, and now will have to maintain yet another syntax
> forever (since of course the old syntax won't die in the next several
> years).
>
Maybe this time we should clearly document the testsuite summary format?
Or even offer a small script to extract it from the "make check" output,
so that those project won't bash us too hard? (Note that I have written
a rough version of such a script in my patch, for use in our own test
cases; but it can't handle multiple summaries in the same output though).
> > Subject: [PATCH] parallel-tests: simplify testsuite summary
> >
> > Prefer a more deterministic, "tabular" format for the testsuite
> > summary, always listing the numbers of passed, failed, xfailed,
> > xpassed, skipped and errored tests, even when these numbers are
> > zero. This simplify the logic of testsuite summary creation,
> > makes it more easily machine-parseable, and will probably allow
> > for easier addition of new kinds of test results.
> >
> > * lib/am/check.am (am__tty_colors_dummy): New make variable, to
> > reduce code duplication. Extracted from previous versions of
> > $(am__tty_colors), and extended by defining two new variables
> > `$mgn' and `$brg'.
> > [%?COLOR%, %!?COLOR%] (am__tty_colors): Use that new variable.
> > (am__text_box): Delete, is not needed anymore.
> > ($(TEST_SUITE_LOG)): Rewrite associated rules to implement the
> > new testsuite summary format.
> > * NEWS: Update.
> > * tests/check10.test: Don't run with the parallel-tests harness
> > too, that makes no sense anymore.
> > * tests/color.test: Update and adjust.
> > * tests/color2.test: Likewise.
> > * tests/parallel-tests.test: Likewise.
> > * tests/parallel-tests6.test: Likewise.
> > * tests/parallel-tests9.test: Likewise.
> > * tests/parallel-tests-unreadable-log.test: Likewise.
> > * tests/parallel-tests-empty-testlogs.test: Likewise.
> > * tests/parallel-tests-log-override-recheck.test: Likewise.
> > * tests/parallel-tests-no-spurious-summary.test: Likewise.
> > * tests/test-driver-custom-multitest.test: Likewise.
> > * tests/test-driver-end-test-results.test: Likewise.
> > * tests/parallel-tests-no-color-in-log.test: New test.
> > * tests/testsuite-summary-color.test: Likewise.
> > * tests/testsuite-summary-count.test: Likewise.
> > * tests/testsuite-summary-count-many.test: Likewise.
> > * tests/testsuite-summary-reference-log.test: Likewise.
> > * tests/testsuite-summary-checks.sh: New auxiliary script, used
> > by the new tests above.
> > * tests/extract-testsuite-summary: Likewise.
> > * tests/trivial-test-driver: Optimize for speed when there are
> > lots of of tests.
> > * tests/Makefile.am (EXTRA_DIST): Distribute them.
> > (testsuite-summary-color.log, testsuite-summary-count.log): Depend
> > on them.
> > (testsuite-summary-count-many.log): Depend on the auxiliary scripts
> > 'trivial-test-driver' and 'extract-testsuite-summary'.
> > (TESTS): Update.
>
>
> > +## When writing the test summary to the console, we want to color a line
> > +## reporting the count of a kind of result *only* if at least one test
>
> s/a kind of/some/
>
Fixed.
> > +## experienced such a result. This function is handy in this regard.
> > + result_count () \
> > + { \
> > + if test x"$$1" = x"--color"; then \
> > + colorize=yes; \
> > + elif test x"$$1" = x"--no-color"; then \
>
> What if output doesn't go to a terminal? Does --no-color get set then,
> or colorize?
>
In that case, `$colorize' is set to "yes", but the variables that should
hold the color escape sequences ($red, $grn, etc.) are set to the empty
string (by the code in $(am__tty_clors)), unless the user explicitly
required otherwise by setting `AM_COLOR_TESTS' to "always". This is the
same "colorizing logic" that is supported for the testsuite progress
output.
And now that I've been forced to spell that out, I see that the `--color'
option name is poorly chosen; I'd say we change it to `--maybe-color',
which is more faithful to the intended seantics. OK? I've done that
in the attached squash-in.
> > + colorize=no; \
> > + else \
> > + echo "$@: invalid 'result_count' usage" >&2; exit 4; \
> > + fi; \
> > + shift; \
> > + desc=$$1 count=$$2; \
> > + if test $$colorize = yes && test $$count -gt 0; then \
> > + color_start=$$3 color_end=$$std; \
> > + else \
> > + color_start= color_end=; \
> > + fi; \
> > + echo "$${color_start}# $$desc $$count$${color_end}"; \
> > + }; \
> > +## A shell function that creates the testsuite summary. We need it
> > +## because we have to create *two* summaries, one for test-suite.log,
> > +## and a possibly-colorized one for console output.
> > + create_testsuite_report () \
> > + { \
> > + result_count $$1 "tests:" $$all "$$brg"; \
> > + result_count $$1 "pass: " $$pass "$$grn"; \
> > + result_count $$1 "skip: " $$skip "$$blu"; \
> > + result_count $$1 "xfail:" $$xfail "$$lgn"; \
> > + result_count $$1 "fail: " $$fail "$$red"; \
> > + result_count $$1 "xpass:" $$xpass "$$red"; \
> > + result_count $$1 "error:" $$error "$$mgn"; \
>
> You don't want to color the actual numbers nor the 'fail:'?
>
Huh? The whole lines get colored ... am I misunderstanding your question?
> Why the shift in capitalization?
>
No good reason at all.
> I know this is totally bike shedding, but see above for rationale.
>
Good point. I'll capitalize the result names ("pass" -> "PASS", etc.),
and adjust the test cases accordingly.
Post Scriptum: I find the summary even nicer with the capitalization
you suggested :-)
> > ## Write "global" testsuite log.
> > { \
> > echo "$(PACKAGE_STRING): $(subdir)/$(TEST_SUITE_LOG)" | \
> > $(am__rst_title); \
> > - echo "$$msg"; \
> > + create_testsuite_report --no-color; \
> > echo; \
> > echo ".. contents:: :depth: 2"; \
> > echo; \
> > @@ -243,22 +236,29 @@ $(TEST_SUITE_LOG): $(TEST_LOGS)
> > done; \
> > } >$(TEST_SUITE_LOG).tmp; \
> > mv $(TEST_SUITE_LOG).tmp $(TEST_SUITE_LOG); \
> > - if test "$$failures" -ne 0; then \
> > - msg="$${msg}See $(subdir)/$(TEST_SUITE_LOG). "; \
> > - if test -n "$(PACKAGE_BUGREPORT)"; then \
> > - msg="$${msg}Please report to $(PACKAGE_BUGREPORT). "; \
> > - fi; \
> > - fi; \
> > - test x"$$VERBOSE" = x || $$exit || cat $(TEST_SUITE_LOG); \
> > -## Emit the test summary on the console, and exit.
> > - $(am__tty_colors); \
> > - if $$exit; then \
> > +## Emit the test summary on the console.
> > + if $$success; then \
> > col="$$grn"; \
> > else \
> > col="$$red"; \
> > + test x"$$VERBOSE" = x || cat $(TEST_SUITE_LOG); \
> > + fi; \
> > +## Multi line coloring is problematic with "less -R", so we really need
> > +## to color each line individually.
> > + echo "$${col}$$br$${std}"; \
> > + echo "$${col}Testsuite summary for $(PACKAGE_STRING)$${std}"; \
> > + echo "$${col}$$br$${std}"; \
> > + create_testsuite_report --color; \
> > + echo "$$col$$br$$std"; \
> > + if $$success; then :; else \
> > + echo "$${col}See $(subdir)/$(TEST_SUITE_LOG)$${std}"; \
> > + if test -n "$(PACKAGE_BUGREPORT)"; then \
> > + echo "$${col}Please report to $(PACKAGE_BUGREPORT)$${std}"; \
> > + fi; \
> > + echo "$$col$$br$$std"; \
> > fi; \
> > - echo "$$msg" | $(am__text_box) "col=$$col" "std=$$std"; \
> > - $$exit
> > +## Be sure to exit with the proper exit status.
> > + $$success
>
> Need to test this on Tru64 and OpenBSD shells. We've been burned with
> false negatives before.
>
I can do the test on OpenBSD 4.6 (not right now, but soonish). I've no
access to True64 systems though.
> > --- /dev/null
> > +++ b/tests/extract-testsuite-summary
> > @@ -0,0 +1,15 @@
> > +#! /usr/bin/env perl
>
> Is it necessary to add another perl script?
>
Not strictly, but it's easier and reasonably fast, and since we can
assume the availability of perl when running the automake testsuite,
what is the problem?
> If yes, why use a different portability mechanism than elsewhere in
> the code base?
>
I don't understand the question, sorry.
> > +# Extract the testsuite summary generated by the parallel-tests harness
> > +# from the output of "make check".
> > +
> > +use warnings FATAL => 'all';
> > +use strict;
> > +
> > +my $br = '=' x 76;
> > +my @sections = ('');
> > +while (<>)
> > + {
> > + push @sections, $_, '' if /$br/;
> > + $sections[-1] .= $_ if !/$br/;
> > + }
> > +print @sections[1..$#sections-1];
>
> > --- /dev/null
> > +++ b/tests/parallel-tests-no-color-in-log.test
> > @@ -0,0 +1,66 @@
>
> > +# Check the testsuite summary with the parallel-tests harness. This
> > +# script is meant to be sourced by other test script, so that it can
>
> s/meant to be /
>
> > +# be used to check different scenarios (color and non-color tests
> > +
This first part of the header was actually spurious (a copy & paste
leftover); sorry about that. I've removed it now.
> > +# Colorized output from testsuite report shouldn't end up into log files.
>
> s/from/& the/
> s/into/in/
>
Fixed.
> > +parallel_tests=yes
> > +. ./defs || Exit 1
> > +
> > +esc=''
> > +
> > +# Check that grep can parse nonprinting characters.
> > +# BSD 'grep' works from a pipe, but not a seekable file.
> > +# GNU or BSD 'grep -a' works on files, but is not portable.
>
> factor into defs?
>
Absolutely... but in a follow-up patch.
> > +case `echo "$esc" | $FGREP "$esc"` in
> > + "$esc") ;;
> > + *) echo "$me: fgrep can't parse nonprinting characters" >&2; Exit 77;;
> > +esac
> > +
> > +TERM=ansi; export TERM
>
> [SNIP]
>
>
> > --- /dev/null
> > +++ b/tests/testsuite-summary-checks.sh
> > @@ -0,0 +1,106 @@
>
> > +# Check the testsuite summary with the parallel-tests harness. This
> > +# script is meant to be sourced by other test script, so that it can
> > +# be used to check different scenarios (colorized and non-colorized
> > +# testsuite output, packages with and without bug-report addresses,
> > +# test scripts in subdirectories, ...)
>
> > +# Quite complexish, but allow the tests in client scripts to be written
> > +# in a "data-driven fashion".
>
> s/ish//
>
> > +do_check ()
> > +{
> > + cat > summary.exp
> > + expect_failure=false
> > + xfail_tests=''
> > + tests="TESTS='$*'"
> > + for t in $*; do
> > + case $t in fail*|xpass*|error*) expect_failure=:;; esac
> > + case $t in xfail*|xpass*) xfail_tests="$xfail_tests $t";; esac
> > + done
> > + test -z "$xfail_tests" || xfail_tests="XFAIL_TESTS='$xfail_tests'"
> > + st=0
> > + eval "env $tests $xfail_tests \$MAKE -e check > stdout || st=\$?"
> > + cat stdout
> > + if $expect_failure; then
> > + test $st -gt 0 || Exit 1
> > + else
> > + test $st -eq 0 || Exit 1
> > + fi
> > + $PERL -w "$testsrcdir"/extract-testsuite-summary stdout > summary.got \
> > + || fatal_ "cannot extract testsuite summary"
> > + cat summary.exp
> > + cat summary.got
> > + if test $use_colors = yes; then
> > + # Use cmp, not diff, because the files might contain binary data.
> > + compare=cmp
> > + else
> > + compare=diff
> > + fi
> > + $compare summary.exp summary.got || Exit 1
> > +}
> > +
> > +br='============================================================================'
> > +
> > +$ACLOCAL
> > +$AUTOCONF
> > +$AUTOMAKE --add-missing
>
> > --- /dev/null
> > +++ b/tests/testsuite-summary-color.test
>
> > +# Check colorization of the testsuite summary.
>
> coloring?
>
OK.
>
> [SNIP]
>
> > --- /dev/null
> > +++ b/tests/testsuite-summary-count.test
> > @@ -0,0 +1,176 @@
>
> > +# Check test counts in the testsuite summary.
> > +
> > +. ./defs-static || Exit 1
> > +
> > +use_colors=no
> > +use_vpath=no
> > +
> > +. "$testsrcdir"/testsuite-summary-checks.sh || Exit 99
> > +
> > +seq_ ()
> > +{
> > + case $# in
> > + 2) l=$1 u=$2;;
> > + *) fatal_ "incorrect usage of 'seq_' function";;
> > + esac
> > + seq $1 $2 || {
> > + i=$l
> > + while test $i -le $u; do
> > + echo $i
> > + i=`expr $i + 1`
> > + done
> > + }
> > +}
> > +
> > +./configure
> > +
> > +header="\
> > +${br}
> > +Testsuite summary for GNU AutoFoo 7.1
> > +${br}"
> > +
> > +success_footer=${br}
> > +
> > +failure_footer="\
> > +${br}
> > +See ./test-suite.log
> > +Please report to address@hidden
>
> Please don't list bad but valid email addresses. Either use
> bug-autoconf or something in the example.com domain. These addresses
> otherwise get picked up from the mail archives by spammers.
>
OK, sorry. I'll use "address@hidden" then (and BTW, you
might want to fix a similar blunder in nodep2.test).
> > +${br}"
> > +
> > +# Corner cases.
> > ...
>
> [SNIP]
>
> Thanks,
> Ralf
Attached is what I've squashed in.
Thanks for the review,
Stefano
diff --git a/lib/Automake/tests/Makefile.in b/lib/Automake/tests/Makefile.in
index af2f22d..6139c8b 100644
--- a/lib/Automake/tests/Makefile.in
+++ b/lib/Automake/tests/Makefile.in
@@ -365,16 +365,16 @@ $(TEST_SUITE_LOG): $(TEST_LOGS)
br='==================='; br=$$br$$br$$br$$br; \
result_count () \
{ \
- if test x"$$1" = x"--color"; then \
- colorize=yes; \
+ if test x"$$1" = x"--maybe-color"; then \
+ maybe_colorize=yes; \
elif test x"$$1" = x"--no-color"; then \
- colorize=no; \
+ maybe_colorize=no; \
else \
echo "$@: invalid 'result_count' usage" >&2; exit 4; \
fi; \
shift; \
desc=$$1 count=$$2; \
- if test $$colorize = yes && test $$count -gt 0; then \
+ if test $$maybe_colorize = yes && test $$count -gt 0; then \
color_start=$$3 color_end=$$std; \
else \
color_start= color_end=; \
@@ -383,13 +383,13 @@ $(TEST_SUITE_LOG): $(TEST_LOGS)
}; \
create_testsuite_report () \
{ \
- result_count $$1 "tests:" $$all "$$brg"; \
- result_count $$1 "pass: " $$pass "$$grn"; \
- result_count $$1 "skip: " $$skip "$$blu"; \
- result_count $$1 "xfail:" $$xfail "$$lgn"; \
- result_count $$1 "fail: " $$fail "$$red"; \
- result_count $$1 "xpass:" $$xpass "$$red"; \
- result_count $$1 "error:" $$error "$$mgn"; \
+ result_count $$1 "TOTAL:" $$all "$$brg"; \
+ result_count $$1 "PASS: " $$pass "$$grn"; \
+ result_count $$1 "SKIP: " $$skip "$$blu"; \
+ result_count $$1 "XFAIL:" $$xfail "$$lgn"; \
+ result_count $$1 "FAIL: " $$fail "$$red"; \
+ result_count $$1 "XPASS:" $$xpass "$$red"; \
+ result_count $$1 "ERROR:" $$error "$$mgn"; \
}; \
{ \
echo "$(PACKAGE_STRING): $(subdir)/$(TEST_SUITE_LOG)" | \
@@ -416,7 +416,7 @@ $(TEST_SUITE_LOG): $(TEST_LOGS)
echo "$${col}$$br$${std}"; \
echo "$${col}Testsuite summary for $(PACKAGE_STRING)$${std}"; \
echo "$${col}$$br$${std}"; \
- create_testsuite_report --color; \
+ create_testsuite_report --maybe-color; \
echo "$$col$$br$$std"; \
if $$success; then :; else \
echo "$${col}See $(subdir)/$(TEST_SUITE_LOG)$${std}"; \
diff --git a/lib/am/check.am b/lib/am/check.am
index 82c60fc..73306bc 100644
--- a/lib/am/check.am
+++ b/lib/am/check.am
@@ -184,20 +184,20 @@ $(TEST_SUITE_LOG): $(TEST_LOGS)
## enclose the testsuite summary report when displayed on the console.
br='==================='; br=$$br$$br$$br$$br; \
## When writing the test summary to the console, we want to color a line
-## reporting the count of a kind of result *only* if at least one test
+## reporting the count of some result *only* if at least one test
## experienced such a result. This function is handy in this regard.
result_count () \
{ \
- if test x"$$1" = x"--color"; then \
- colorize=yes; \
+ if test x"$$1" = x"--maybe-color"; then \
+ maybe_colorize=yes; \
elif test x"$$1" = x"--no-color"; then \
- colorize=no; \
+ maybe_colorize=no; \
else \
echo "$@: invalid 'result_count' usage" >&2; exit 4; \
fi; \
shift; \
desc=$$1 count=$$2; \
- if test $$colorize = yes && test $$count -gt 0; then \
+ if test $$maybe_colorize = yes && test $$count -gt 0; then \
color_start=$$3 color_end=$$std; \
else \
color_start= color_end=; \
@@ -209,13 +209,13 @@ $(TEST_SUITE_LOG): $(TEST_LOGS)
## and a possibly-colorized one for console output.
create_testsuite_report () \
{ \
- result_count $$1 "tests:" $$all "$$brg"; \
- result_count $$1 "pass: " $$pass "$$grn"; \
- result_count $$1 "skip: " $$skip "$$blu"; \
- result_count $$1 "xfail:" $$xfail "$$lgn"; \
- result_count $$1 "fail: " $$fail "$$red"; \
- result_count $$1 "xpass:" $$xpass "$$red"; \
- result_count $$1 "error:" $$error "$$mgn"; \
+ result_count $$1 "TOTAL:" $$all "$$brg"; \
+ result_count $$1 "PASS: " $$pass "$$grn"; \
+ result_count $$1 "SKIP: " $$skip "$$blu"; \
+ result_count $$1 "XFAIL:" $$xfail "$$lgn"; \
+ result_count $$1 "FAIL: " $$fail "$$red"; \
+ result_count $$1 "XPASS:" $$xpass "$$red"; \
+ result_count $$1 "ERROR:" $$error "$$mgn"; \
}; \
## Write "global" testsuite log.
{ \
@@ -248,7 +248,8 @@ $(TEST_SUITE_LOG): $(TEST_LOGS)
echo "$${col}$$br$${std}"; \
echo "$${col}Testsuite summary for $(PACKAGE_STRING)$${std}"; \
echo "$${col}$$br$${std}"; \
- create_testsuite_report --color; \
+## This is expected to go to the console, so it might have to be colorized.
+ create_testsuite_report --maybe-color; \
echo "$$col$$br$$std"; \
if $$success; then :; else \
echo "$${col}See $(subdir)/$(TEST_SUITE_LOG)$${std}"; \
diff --git a/tests/Makefile.in b/tests/Makefile.in
index 7f66594..c72b1cf 100644
--- a/tests/Makefile.in
+++ b/tests/Makefile.in
@@ -1423,16 +1423,16 @@ $(TEST_SUITE_LOG): $(TEST_LOGS)
br='==================='; br=$$br$$br$$br$$br; \
result_count () \
{ \
- if test x"$$1" = x"--color"; then \
- colorize=yes; \
+ if test x"$$1" = x"--maybe-color"; then \
+ maybe_colorize=yes; \
elif test x"$$1" = x"--no-color"; then \
- colorize=no; \
+ maybe_colorize=no; \
else \
echo "$@: invalid 'result_count' usage" >&2; exit 4; \
fi; \
shift; \
desc=$$1 count=$$2; \
- if test $$colorize = yes && test $$count -gt 0; then \
+ if test $$maybe_colorize = yes && test $$count -gt 0; then \
color_start=$$3 color_end=$$std; \
else \
color_start= color_end=; \
@@ -1441,13 +1441,13 @@ $(TEST_SUITE_LOG): $(TEST_LOGS)
}; \
create_testsuite_report () \
{ \
- result_count $$1 "tests:" $$all "$$brg"; \
- result_count $$1 "pass: " $$pass "$$grn"; \
- result_count $$1 "skip: " $$skip "$$blu"; \
- result_count $$1 "xfail:" $$xfail "$$lgn"; \
- result_count $$1 "fail: " $$fail "$$red"; \
- result_count $$1 "xpass:" $$xpass "$$red"; \
- result_count $$1 "error:" $$error "$$mgn"; \
+ result_count $$1 "TOTAL:" $$all "$$brg"; \
+ result_count $$1 "PASS: " $$pass "$$grn"; \
+ result_count $$1 "SKIP: " $$skip "$$blu"; \
+ result_count $$1 "XFAIL:" $$xfail "$$lgn"; \
+ result_count $$1 "FAIL: " $$fail "$$red"; \
+ result_count $$1 "XPASS:" $$xpass "$$red"; \
+ result_count $$1 "ERROR:" $$error "$$mgn"; \
}; \
{ \
echo "$(PACKAGE_STRING): $(subdir)/$(TEST_SUITE_LOG)" | \
@@ -1474,7 +1474,7 @@ $(TEST_SUITE_LOG): $(TEST_LOGS)
echo "$${col}$$br$${std}"; \
echo "$${col}Testsuite summary for $(PACKAGE_STRING)$${std}"; \
echo "$${col}$$br$${std}"; \
- create_testsuite_report --color; \
+ create_testsuite_report --maybe-color; \
echo "$$col$$br$$std"; \
if $$success; then :; else \
echo "$${col}See $(subdir)/$(TEST_SUITE_LOG)$${std}"; \
diff --git a/tests/color.test b/tests/color.test
index 73534d1..b2c24a4 100755
--- a/tests/color.test
+++ b/tests/color.test
@@ -108,15 +108,15 @@ test_no_color ()
# colorized.
{
# Not a useless use of cat; see above comments about grep.
- cat stdout | grep -i "tests.*:"
- cat stdout | grep -i "pass.*:"
- cat stdout | grep -i "fail.*:"
- cat stdout | grep -i "skip.*:"
- cat stdout | grep -i "xfail.*:"
- cat stdout | grep -i "xpass.*:"
- cat stdout | grep -i "error.*:"
- cat stdout | grep -i 'test.*expected'
- cat stdout | grep -i 'test.*not run'
+ cat stdout | grep "TOTAL.*:"
+ cat stdout | grep "PASS.*:"
+ cat stdout | grep "FAIL.*:"
+ cat stdout | grep "SKIP.*:"
+ cat stdout | grep "XFAIL.*:"
+ cat stdout | grep "XPASS.*:"
+ cat stdout | grep "ERROR.*:"
+ cat stdout | grep 'test.*expected'
+ cat stdout | grep 'test.*not run'
cat stdout | grep '===='
cat stdout | grep '[Ss]ee .*test-suite\.log'
cat stdout | grep '[Tt]estsuite summary'
diff --git a/tests/color2.test b/tests/color2.test
index 27093c6..26979db 100755
--- a/tests/color2.test
+++ b/tests/color2.test
@@ -121,15 +121,15 @@ test_no_color ()
# colorized.
{
# Not a useless use of cat; see above comments about grep.
- cat stdout | grep -i "tests.*:"
- cat stdout | grep -i "pass.*:"
- cat stdout | grep -i "fail.*:"
- cat stdout | grep -i "skip.*:"
- cat stdout | grep -i "xfail.*:"
- cat stdout | grep -i "xpass.*:"
- cat stdout | grep -i "error.*:"
- cat stdout | grep -i 'test.*expected'
- cat stdout | grep -i 'test.*not run'
+ cat stdout | grep "TOTAL.*:"
+ cat stdout | grep "PASS.*:"
+ cat stdout | grep "FAIL.*:"
+ cat stdout | grep "SKIP.*:"
+ cat stdout | grep "XFAIL.*:"
+ cat stdout | grep "XPASS.*:"
+ cat stdout | grep "ERROR.*:"
+ cat stdout | grep 'test.*expected'
+ cat stdout | grep 'test.*not run'
cat stdout | grep '===='
cat stdout | grep '[Ss]ee .*test-suite\.log'
cat stdout | grep '[Tt]estsuite summary'
diff --git a/tests/parallel-tests-empty-testlogs.test
b/tests/parallel-tests-empty-testlogs.test
index e91b7b6..e3d77ea 100755
--- a/tests/parallel-tests-empty-testlogs.test
+++ b/tests/parallel-tests-empty-testlogs.test
@@ -55,7 +55,7 @@ $AUTOMAKE -a
no_test_has_run ()
{
ls -1 *.log | grep -v '^test-suite\.log$' | grep . && Exit 1
- grep '^# tests: *0$' test-suite.log
+ grep '^# TOTAL: *0$' test-suite.log
:
}
diff --git a/tests/parallel-tests-log-override-recheck.test
b/tests/parallel-tests-log-override-recheck.test
index d68cdf5..613bea1 100755
--- a/tests/parallel-tests-log-override-recheck.test
+++ b/tests/parallel-tests-log-override-recheck.test
@@ -65,9 +65,9 @@ TEST_SUITE_LOG=my.log $MAKE -e recheck >stdout \
&& { cat stdout; Exit 1; }
cat stdout
ls -l
-grep '# tests: *2$' stdout
-grep '# fail: *1$' stdout
-grep '# error: *1$' stdout
+grep '^# TOTAL: *2$' stdout
+grep '^# FAIL: *1$' stdout
+grep '^# ERROR: *1$' stdout
for x in stdout my.log; do
$FGREP foo.test $x && Exit 1
$FGREP bar.test $x
@@ -79,9 +79,9 @@ BAZ_EXIT_STATUS=0 TEST_SUITE_LOG=my2.log $MAKE -e recheck
>stdout \
&& { cat stdout; Exit 1; }
cat stdout
ls -l
-grep '# tests: *2$' stdout
-grep '# pass: *1$' stdout
-grep '# error: *1$' stdout
+grep '^# TOTAL: *2$' stdout
+grep '^# PASS: *1$' stdout
+grep '^# ERROR: *1$' stdout
$FGREP foo.test stdout && Exit 1
$FGREP bar.test stdout
$FGREP baz.test stdout
diff --git a/tests/parallel-tests-no-color-in-log.test
b/tests/parallel-tests-no-color-in-log.test
index fd7435e..ab3f371 100755
--- a/tests/parallel-tests-no-color-in-log.test
+++ b/tests/parallel-tests-no-color-in-log.test
@@ -14,11 +14,7 @@
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
-# Check the testsuite summary with the parallel-tests harness. This
-# script is meant to be sourced by other test script, so that it can
-# be used to check different scenarios (color and non-color tests
-
-# Colorized output from testsuite report shouldn't end up into log files.
+# Colorized output from the testsuite report shouldn't end up in log files.
parallel_tests=yes
. ./defs || Exit 1
diff --git a/tests/parallel-tests-no-spurious-summary.test
b/tests/parallel-tests-no-spurious-summary.test
index f016d96..7fb95f5 100755
--- a/tests/parallel-tests-no-spurious-summary.test
+++ b/tests/parallel-tests-no-spurious-summary.test
@@ -62,9 +62,9 @@ grep '^:test-result: SKIP$' foo.log
grep '^:test-result:FAIL$' bar.log
grep '^:test-result: XPASS$' bar.log
-grep '^# tests: *2$' output
-grep '^# pass: *2$' output
-for result in fail xfail xpass skip error; do
+grep '^# TOTAL: *2$' output
+grep '^# PASS: *2$' output
+for result in FAIL XFAIL XPASS SKIP ERROR; do
grep "^# $result: *0$" output
done
diff --git a/tests/parallel-tests-unreadable-log.test
b/tests/parallel-tests-unreadable-log.test
index ca7fe2b..e8e101b 100755
--- a/tests/parallel-tests-unreadable-log.test
+++ b/tests/parallel-tests-unreadable-log.test
@@ -54,7 +54,7 @@ chmod a-r foo.log bar.log
test ! -r foo.log || Exit 77
$MAKE test-suite.log >stdout && { cat stdout; Exit 1; }
cat stdout
-grep '^# error: *2$' stdout
-grep '^# error: *2$' test-suite.log
+grep '^# ERROR: *2$' stdout
+grep '^# ERROR: *2$' test-suite.log
:
diff --git a/tests/parallel-tests.test b/tests/parallel-tests.test
index 5f72f5a..44e8846 100755
--- a/tests/parallel-tests.test
+++ b/tests/parallel-tests.test
@@ -111,9 +111,9 @@ test -f foo.log
grep foo.test stdout
grep bar.test stdout && Exit 1
grep baz.test stdout && Exit 1
-grep '^# pass: *1$' stdout
-grep '^# fail: *1$' stdout
-grep '^# error: *1$' stdout
+grep '^# PASS: *1$' stdout
+grep '^# FAIL: *1$' stdout
+grep '^# ERROR: *1$' stdout
# Now, explicitly retry with all test logs already updated, and ensure
# that the summary is still displayed.
@@ -122,9 +122,9 @@ cat stdout
grep foo.test stdout && Exit 1
grep bar.test stdout && Exit 1
grep baz.test stdout && Exit 1
-grep '^# pass: *1$' stdout
-grep '^# fail: *1$' stdout
-grep '^# error: *1$' stdout
+grep '^# PASS: *1$' stdout
+grep '^# FAIL: *1$' stdout
+grep '^# ERROR: *1$' stdout
# Lazily rerunning only foo should only rerun this one test.
env RECHECK_LOGS=foo.log $MAKE -e check > stdout && { cat stdout; Exit 1; }
@@ -132,9 +132,9 @@ cat stdout
grep foo.test stdout
grep bar.test stdout && Exit 1
grep baz.test stdout && Exit 1
-grep '^# pass: *1$' stdout
-grep '^# fail: *1$' stdout
-grep '^# error: *1$' stdout
+grep '^# PASS: *1$' stdout
+grep '^# FAIL: *1$' stdout
+grep '^# ERROR: *1$' stdout
# Test VERBOSE.
env VERBOSE=yes $MAKE -e check > stdout && { cat stdout; Exit 1; }
diff --git a/tests/parallel-tests6.test b/tests/parallel-tests6.test
index 9070192..e0c535e 100755
--- a/tests/parallel-tests6.test
+++ b/tests/parallel-tests6.test
@@ -36,7 +36,7 @@ $AUTOMAKE -a
./configure
$MAKE check >stdout || { cat stdout; Exit 1; }
cat stdout
-for x in tests pass fail xpass fail skip error; do
+for x in TOTAL PASS FAIL XPASS FAIL SKIP ERROR; do
grep "^# $x: *0$" stdout
done
diff --git a/tests/parallel-tests9.test b/tests/parallel-tests9.test
index ffdfef5..2a6922e 100755
--- a/tests/parallel-tests9.test
+++ b/tests/parallel-tests9.test
@@ -58,18 +58,18 @@ $AUTOMAKE -a
./configure
$MAKE check >stdout && { cat stdout; Exit 1; }
cat stdout
-grep '^# pass: *1$' stdout
-grep '^# fail: *1$' stdout
-grep '^# error: *1$' stdout
+grep '^# PASS: *1$' stdout
+grep '^# FAIL: *1$' stdout
+grep '^# ERROR: *1$' stdout
$MAKE recheck >stdout && { cat stdout; Exit 1; }
cat stdout
grep foo.test stdout && Exit 1
grep bar.test stdout || Exit 1
grep baz.test stdout || Exit 1
-grep '^# pass: *0$' stdout
-grep '^# fail: *1$' stdout
-grep '^# error: *1$' stdout
+grep '^# PASS: *0$' stdout
+grep '^# FAIL: *1$' stdout
+grep '^# ERROR: *1$' stdout
# If we cannot read the log file, then redo it as well.
chmod a-r foo.log
diff --git a/tests/test-driver-custom-multitest.test
b/tests/test-driver-custom-multitest.test
index 797572d..0316f0f 100755
--- a/tests/test-driver-custom-multitest.test
+++ b/tests/test-driver-custom-multitest.test
@@ -162,13 +162,13 @@ for vpath in : false; do
grep "^ERROR: $tst\.t, testcase 6" stdout
# Check counts of testcases in testsuite summary printed on console.
- grep "^# tests: *$expected_tests$" stdout
- grep "^# pass: *$expected_pass$" stdout
- grep "^# xpass: *$expected_xpass$" stdout
- grep "^# fail: *$expected_fail$" stdout
- grep "^# xfail: *$expected_xfail$" stdout
- grep "^# skip: *$expected_skip$" stdout
- grep "^# error: *$expected_error$" stdout
+ grep "^# TOTAL: *$expected_tests$" stdout
+ grep "^# PASS: *$expected_pass$" stdout
+ grep "^# XPASS: *$expected_xpass$" stdout
+ grep "^# FAIL: *$expected_fail$" stdout
+ grep "^# XFAIL: *$expected_xfail$" stdout
+ grep "^# SKIP: *$expected_skip$" stdout
+ grep "^# ERROR: *$expected_error$" stdout
# Check that the content of, and only of, the test logs with at least
# one failing test case has been copied into `test-suite.log'. Note
@@ -193,13 +193,13 @@ for vpath in : false; do
test `grep -c '^XFAIL:' stdout` -eq 1
$EGREP '^(FAIL|XPASS|ERROR)' stdout && Exit 1
# Check counts of testcases in testsuite summary printed on console.
- grep "^# tests: *7$" stdout
- grep "^# pass: *4$" stdout
- grep "^# xpass: *0$" stdout
- grep "^# fail: *0$" stdout
- grep "^# xfail: *1$" stdout
- grep "^# skip: *2$" stdout
- grep "^# error: *0$" stdout
+ grep "^# TOTAL: *7$" stdout
+ grep "^# PASS: *4$" stdout
+ grep "^# XPASS: *0$" stdout
+ grep "^# FAIL: *0$" stdout
+ grep "^# XFAIL: *1$" stdout
+ grep "^# SKIP: *2$" stdout
+ grep "^# ERROR: *0$" stdout
cd $srcdir
diff --git a/tests/test-driver-end-test-results.test
b/tests/test-driver-end-test-results.test
index f864bc2..7008700 100755
--- a/tests/test-driver-end-test-results.test
+++ b/tests/test-driver-end-test-results.test
@@ -85,10 +85,10 @@ $MAKE check >output 2>&1 || { cat output; Exit 1; }
cat output
cat a.log; cat b.log; cat c.log; cat d.log # For debugging.
-grep '^# tests: *4$' output
-grep '^# pass: *3$' output
-grep '^# skip: *1$' output
-for result in xpass fail xfail error; do
+grep '^# TOTAL: *4$' output
+grep '^# PASS: *3$' output
+grep '^# SKIP: *1$' output
+for result in XPASS FAIL XFAIL ERROR; do
grep "^# $result: *0$" output
done
diff --git a/tests/testsuite-summary-checks.sh
b/tests/testsuite-summary-checks.sh
index 252b4e6..f318d75 100755
--- a/tests/testsuite-summary-checks.sh
+++ b/tests/testsuite-summary-checks.sh
@@ -18,7 +18,7 @@
# script is meant to be sourced by other test script, so that it can
# be used to check different scenarios (colorized and non-colorized
# testsuite output, packages with and without bug-report addresses,
-# test scripts in subdirectories, ...)
+# testsuites in subdirectories, ...)
parallel_tests=yes
. ./defs || Exit 1
@@ -37,7 +37,7 @@ case $use_colors in
esac
cat > configure.in <<END
-AC_INIT([GNU AutoFoo], [7.1], address@hidden)
+AC_INIT([GNU AutoFoo], [7.1], address@hidden)
AM_INIT_AUTOMAKE([$am_opts])
AC_CONFIG_FILES([Makefile])
AC_OUTPUT
diff --git a/tests/testsuite-summary-color.test
b/tests/testsuite-summary-color.test
index d754aff..3ac9544 100755
--- a/tests/testsuite-summary-color.test
+++ b/tests/testsuite-summary-color.test
@@ -14,7 +14,7 @@
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
-# Check colorization of the testsuite summary.
+# Check coloring of the testsuite summary.
. ./defs-static || Exit 1
@@ -49,150 +49,150 @@ ${red}${br}${std}"
failure_footer="\
${red}${br}${std}
${red}See ./test-suite.log${std}
-${red}Please report to address@hidden
+${red}Please report to address@hidden
${red}${br}${std}"
do_check '' <<END
$success_header
-# tests: 0
-# pass: 0
-# skip: 0
-# xfail: 0
-# fail: 0
-# xpass: 0
-# error: 0
+# TOTAL: 0
+# PASS: 0
+# SKIP: 0
+# XFAIL: 0
+# FAIL: 0
+# XPASS: 0
+# ERROR: 0
$success_footer
END
do_check pass.t <<END
$success_header
-${brg}# tests: 1${std}
-${grn}# pass: 1${std}
-# skip: 0
-# xfail: 0
-# fail: 0
-# xpass: 0
-# error: 0
+${brg}# TOTAL: 1${std}
+${grn}# PASS: 1${std}
+# SKIP: 0
+# XFAIL: 0
+# FAIL: 0
+# XPASS: 0
+# ERROR: 0
$success_footer
END
do_check skip.t <<END
$success_header
-${brg}# tests: 1${std}
-# pass: 0
-${blu}# skip: 1${std}
-# xfail: 0
-# fail: 0
-# xpass: 0
-# error: 0
+${brg}# TOTAL: 1${std}
+# PASS: 0
+${blu}# SKIP: 1${std}
+# XFAIL: 0
+# FAIL: 0
+# XPASS: 0
+# ERROR: 0
$success_footer
END
do_check xfail.t <<END
$success_header
-${brg}# tests: 1${std}
-# pass: 0
-# skip: 0
-${lgn}# xfail: 1${std}
-# fail: 0
-# xpass: 0
-# error: 0
+${brg}# TOTAL: 1${std}
+# PASS: 0
+# SKIP: 0
+${lgn}# XFAIL: 1${std}
+# FAIL: 0
+# XPASS: 0
+# ERROR: 0
$success_footer
END
do_check fail.t <<END
$failure_header
-${brg}# tests: 1${std}
-# pass: 0
-# skip: 0
-# xfail: 0
-${red}# fail: 1${std}
-# xpass: 0
-# error: 0
+${brg}# TOTAL: 1${std}
+# PASS: 0
+# SKIP: 0
+# XFAIL: 0
+${red}# FAIL: 1${std}
+# XPASS: 0
+# ERROR: 0
$failure_footer
END
do_check xpass.t <<END
$failure_header
-${brg}# tests: 1${std}
-# pass: 0
-# skip: 0
-# xfail: 0
-# fail: 0
-${red}# xpass: 1${std}
-# error: 0
+${brg}# TOTAL: 1${std}
+# PASS: 0
+# SKIP: 0
+# XFAIL: 0
+# FAIL: 0
+${red}# XPASS: 1${std}
+# ERROR: 0
$failure_footer
END
do_check error.t <<END
$failure_header
-${brg}# tests: 1${std}
-# pass: 0
-# skip: 0
-# xfail: 0
-# fail: 0
-# xpass: 0
-${mgn}# error: 1${std}
+${brg}# TOTAL: 1${std}
+# PASS: 0
+# SKIP: 0
+# XFAIL: 0
+# FAIL: 0
+# XPASS: 0
+${mgn}# ERROR: 1${std}
$failure_footer
END
do_check pass.t xfail.t skip.t <<END
$success_header
-${brg}# tests: 3${std}
-${grn}# pass: 1${std}
-${blu}# skip: 1${std}
-${lgn}# xfail: 1${std}
-# fail: 0
-# xpass: 0
-# error: 0
+${brg}# TOTAL: 3${std}
+${grn}# PASS: 1${std}
+${blu}# SKIP: 1${std}
+${lgn}# XFAIL: 1${std}
+# FAIL: 0
+# XPASS: 0
+# ERROR: 0
$success_footer
END
do_check pass.t fail.t skip.t <<END
$failure_header
-${brg}# tests: 3${std}
-${grn}# pass: 1${std}
-${blu}# skip: 1${std}
-# xfail: 0
-${red}# fail: 1${std}
-# xpass: 0
-# error: 0
+${brg}# TOTAL: 3${std}
+${grn}# PASS: 1${std}
+${blu}# SKIP: 1${std}
+# XFAIL: 0
+${red}# FAIL: 1${std}
+# XPASS: 0
+# ERROR: 0
$failure_footer
END
do_check pass.t xfail.t xpass.t <<END
$failure_header
-${brg}# tests: 3${std}
-${grn}# pass: 1${std}
-# skip: 0
-${lgn}# xfail: 1${std}
-# fail: 0
-${red}# xpass: 1${std}
-# error: 0
+${brg}# TOTAL: 3${std}
+${grn}# PASS: 1${std}
+# SKIP: 0
+${lgn}# XFAIL: 1${std}
+# FAIL: 0
+${red}# XPASS: 1${std}
+# ERROR: 0
$failure_footer
END
do_check skip.t xfail.t error.t <<END
$failure_header
-${brg}# tests: 3${std}
-# pass: 0
-${blu}# skip: 1${std}
-${lgn}# xfail: 1${std}
-# fail: 0
-# xpass: 0
-${mgn}# error: 1${std}
+${brg}# TOTAL: 3${std}
+# PASS: 0
+${blu}# SKIP: 1${std}
+${lgn}# XFAIL: 1${std}
+# FAIL: 0
+# XPASS: 0
+${mgn}# ERROR: 1${std}
$failure_footer
END
do_check pass.t skip.t xfail.t fail.t xpass.t error.t <<END
$failure_header
-${brg}# tests: 6${std}
-${grn}# pass: 1${std}
-${blu}# skip: 1${std}
-${lgn}# xfail: 1${std}
-${red}# fail: 1${std}
-${red}# xpass: 1${std}
-${mgn}# error: 1${std}
+${brg}# TOTAL: 6${std}
+${grn}# PASS: 1${std}
+${blu}# SKIP: 1${std}
+${lgn}# XFAIL: 1${std}
+${red}# FAIL: 1${std}
+${red}# XPASS: 1${std}
+${mgn}# ERROR: 1${std}
$failure_footer
END
diff --git a/tests/testsuite-summary-count-many.test
b/tests/testsuite-summary-count-many.test
index d5b2a34..b0232c7 100755
--- a/tests/testsuite-summary-count-many.test
+++ b/tests/testsuite-summary-count-many.test
@@ -60,20 +60,20 @@ $PERL -w -e '
# implementation is too slow to handle that :-(
my $base = 5;
my %count = (
- tests => $base * 1000,
- pass => $base * 700,
- skip => $base * 200,
- xfail => $base * 80,
- fail => $base * 10,
- xpass => $base * 7,
- error => $base * 3,
+ TOTAL => $base * 1000,
+ PASS => $base * 700,
+ SKIP => $base * 200,
+ XFAIL => $base * 80,
+ FAIL => $base * 10,
+ XPASS => $base * 7,
+ ERROR => $base * 3,
);
- my @results = qw/pass skip xfail fail xpass error/;
+ my @results = qw/PASS SKIP XFAIL FAIL XPASS ERROR/;
open (RES, ">results.txt") or die "opening results.txt: $!\n";
open (CNT, ">count.txt") or die "opening count.txt: $!\n";
- printf CNT "# %-6s %d\n", "tests:", $count{tests};
+ printf CNT "# %-6s %d\n", "TOTAL:", $count{TOTAL};
for my $res (@results)
{
my $uc_res = uc $res;
diff --git a/tests/testsuite-summary-count.test
b/tests/testsuite-summary-count.test
index 7ef3247..c6be01e 100755
--- a/tests/testsuite-summary-count.test
+++ b/tests/testsuite-summary-count.test
@@ -50,44 +50,44 @@ success_footer=${br}
failure_footer="\
${br}
See ./test-suite.log
-Please report to address@hidden
+Please report to address@hidden
${br}"
# Corner cases.
do_check '' <<END
$header
-# tests: 0
-# pass: 0
-# skip: 0
-# xfail: 0
-# fail: 0
-# xpass: 0
-# error: 0
+# TOTAL: 0
+# PASS: 0
+# SKIP: 0
+# XFAIL: 0
+# FAIL: 0
+# XPASS: 0
+# ERROR: 0
$success_footer
END
do_check pass.t <<END
$header
-# tests: 1
-# pass: 1
-# skip: 0
-# xfail: 0
-# fail: 0
-# xpass: 0
-# error: 0
+# TOTAL: 1
+# PASS: 1
+# SKIP: 0
+# XFAIL: 0
+# FAIL: 0
+# XPASS: 0
+# ERROR: 0
$success_footer
END
do_check fail.t <<END
$header
-# tests: 1
-# pass: 0
-# skip: 0
-# xfail: 0
-# fail: 1
-# xpass: 0
-# error: 0
+# TOTAL: 1
+# PASS: 0
+# SKIP: 0
+# XFAIL: 0
+# FAIL: 1
+# XPASS: 0
+# ERROR: 0
$failure_footer
END
@@ -95,25 +95,25 @@ END
do_check skip.t skip2.t skip3.t xfail.t xfail2.t <<END
$header
-# tests: 5
-# pass: 0
-# skip: 3
-# xfail: 2
-# fail: 0
-# xpass: 0
-# error: 0
+# TOTAL: 5
+# PASS: 0
+# SKIP: 3
+# XFAIL: 2
+# FAIL: 0
+# XPASS: 0
+# ERROR: 0
$success_footer
END
do_check pass.t pass2.t xfail.t xpass.t error.t error2.t <<END
$header
-# tests: 6
-# pass: 2
-# skip: 0
-# xfail: 1
-# fail: 0
-# xpass: 1
-# error: 2
+# TOTAL: 6
+# PASS: 2
+# SKIP: 0
+# XFAIL: 1
+# FAIL: 0
+# XPASS: 1
+# ERROR: 2
$failure_footer
END
@@ -134,13 +134,13 @@ error=`seq_ 1 $error_count | sed 's/.*/error-&.t/'`
do_check $pass $skip $xfail $fail $xpass $error <<END
$header
-# tests: $tests_count
-# pass: $pass_count
-# skip: $skip_count
-# xfail: $xfail_count
-# fail: $fail_count
-# xpass: $xpass_count
-# error: $error_count
+# TOTAL: $tests_count
+# PASS: $pass_count
+# SKIP: $skip_count
+# XFAIL: $xfail_count
+# FAIL: $fail_count
+# XPASS: $xpass_count
+# ERROR: $error_count
$failure_footer
END
@@ -163,13 +163,13 @@ error=`seq_ 1 $error_count | sed 's/.*/error-&.t/'`
do_check $pass $skip $xfail $fail $xpass $error <<END
$header
-# tests: $tests_count
-# pass: $pass_count
-# skip: $skip_count
-# xfail: $xfail_count
-# fail: $fail_count
-# xpass: $xpass_count
-# error: $error_count
+# TOTAL: $tests_count
+# PASS: $pass_count
+# SKIP: $skip_count
+# XFAIL: $xfail_count
+# FAIL: $fail_count
+# XPASS: $xpass_count
+# ERROR: $error_count
$failure_footer
END