[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [GSoC] preliminary tests on TAP support
|
From: |
Stefano Lattarini |
|
Subject: |
Re: [GSoC] preliminary tests on TAP support |
|
Date: |
Sat, 9 Jul 2011 22:44:56 +0200 |
|
User-agent: |
KMail/1.13.3 (Linux/2.6.30-2-686; KDE/4.4.4; i686; ; ) |
On Thursday 07 July 2011, Stefano Lattarini wrote:
> Hello Automakers.
>
> I've created a nice battery of tests on TAP support, and some sketchy
> documentation. Some tests are still incomplete, but I think it's better
> to start getting some early feedback anyway. Thus I've already pushed
> them to the temporary branch 'GSoC/experimental/test-results-work' (note
> that, since that branch is temporary and rewindable, the patch can still
> be amended freely). Attached is what I've pushed.
>
> As usual, reviews, suggestions and criticism are welcome. I should be
> able to finish the incomplete tests by tomorrow evening.
>
> Regards,
> Stefano
>
I've now squashed in the attached patch. Notice that some tests are
still incomplete or buggy, and there are even some incompatibilities
between different tests; but these issues can be sorted out once we
have the experimental/exploratory implementation ready and rolling.
Regards,
Stefano
diff --git a/doc/automake.texi b/doc/automake.texi
index 9344beb..0c75a22 100644
--- a/doc/automake.texi
+++ b/doc/automake.texi
@@ -9352,21 +9352,37 @@ from standard error. Lines written to standard output
matching
lines must not be considered test output.
@end quotation
-Links to more extensive official or third-party documentation and
-resources:
address@hidden
+Here are some links to more extensive official or third-party documentation
+and resources:
@itemize @bullet
@item
-The (mostly) official documentation about the TAP format and protocol:
address@hidden://search.cpan.org/~petdance/Test-Harness/lib/Test/Harness/TAP.pod}
address@hidden://search.cpan.org/~petdance/Test-Harness/lib/Test/Harness/TAP.pod,
+ @samp{Test::Harness::TAP}},
+the (mostly) official documentation about the TAP format and protocol.
address@hidden
address@hidden://search.cpan.org/~andya/Test-Harness/bin/prove,
+ @command{prove}},
+the most famous command-line TAP test driver, included in the distribution
+of @command{perl} and
address@hidden://search.cpan.org/~andya/Test-Harness/lib/Test/Harness.pm,
+ @samp{Test::Harness}}.
address@hidden
+The @uref{http://testanything.org/wiki/index.php/Main_Page,TAP wiki}.
@item
-The most famous command-line TAP test driver, included in the distribution
-of @command{perl} and @samp{Test::Harness}:
address@hidden://search.cpan.org/~andya/Test-Harness/bin/prove}
+A ``gentle introduction'' to testing for perl coders:
address@hidden://search.cpan.org/dist/Test-Simple/lib/Test/Tutorial.pod,
+ @samp{Test::Tutorial}}.
@item
-A wiki devoted to TAP:
address@hidden://testanything.org/wiki/index.php/Main_Page}
address@hidden://search.cpan.org/~mschwern/Test-Simple/lib/Test/Simple.pm,
+ @samp{Test::Simple}}
+and
address@hidden://search.cpan.org/~mschwern/Test-Simple/lib/Test/More.pm,
+ @samp{Test::More}},
+the standard perl testing libraries, which are based on TAP.
@item
address@hidden: links to TAP-producing frameworks and libraries.
address@hidden://www.eyrie.org/~eagle/software/c-tap-harness/,C TAP Harness},
+a C-based project implementing both a TAP producer and a TAP consumer.
@end itemize
Give example of output from @command{prove} command (when used in
@@ -9390,18 +9406,16 @@ character) are copied only in the @file{.log} file.
The option
progress output too (@emph{TODO}: give example). The option
@option{--no-comments} restore the default behaviour.
-Differences and incompatibilities with other TAP parsers and runners:
address@hidden
+Differences and incompatibilities with other TAP parsers and drivers:
@itemize @bullet
@item
-To avoid output synchronization problems, we read TAP directives not only
-from stdout, but also from stderr. This is similar to what happens using
-the @option{--merge} option of @command{prove}. @emph{FIXME}: maybe we
-should provide an option do disable this behaviour? I really think so.
address@hidden
A @code{Bail out!} directive doesn't stop the whole testsuite, but only
the test script it occurs into. This doesn't follows TAP specifications,
but on the other hand maximize compatibility (and code sharing) with
the ``hard error'' concept of the default @option{parallel-tests} driver.
address@hidden
address@hidden: there's surely something else ...
@end itemize
@node DejaGnu Tests
diff --git a/tests/tap-autonumber.test b/tests/tap-autonumber.test
index 6b77484..aedc253 100755
--- a/tests/tap-autonumber.test
+++ b/tests/tap-autonumber.test
@@ -75,11 +75,11 @@ FAIL: all.test 4
FAIL: all.test 5 bar
FAIL: all.test 6 - bar2
PASS: all.test 7
-XPASS: all.test 8
-XFAIL: all.test 9 (who cares?)
+XPASS: all.test 8 # TODO
+XFAIL: all.test 9 # TODO who cares?
PASS: all.test 10
FAIL: all.test 11
-SKIP: all.test 12 muu
+SKIP: all.test 12 muu # SKIP
FAIL: all.test 13
PASS: all.test 14
END
diff --git a/tests/tap-bailout.test b/tests/tap-bailout.test
index 442c274..61b1bfa 100755
--- a/tests/tap-bailout.test
+++ b/tests/tap-bailout.test
@@ -34,7 +34,10 @@ TEST_LOG_COMPILER = cat
TESTS = a.test b.test c.test d.test e.test
END
-# pass += 1, fail +=1, error += 1
+: > exp
+
+#------------------------------------------------------------------
+
cat > a.test <<END
1..4
ok 1
@@ -44,41 +47,80 @@ not ok 3
ok 4 # SKIP
END
-# skip += 1, xpass += 1, xfail += 1, error += 1
+cat >> exp <<END
+PASS: a.test 1
+FAIL: a.test 2
+ERROR: a.test - Bail out!
+END
+
+# pass += 1, fail +=1, error += 1
+
+#------------------------------------------------------------------
+
cat > b.test <<END
1..7
ok 1 # SKIP
ok 2 # TODO
not ok 3 # TODO
-Bail out!${tab}${sp}${sp}${tab}We're out of disk space.
+Bail out!${tab} ${tab}We're out of disk space.
ok 4
not ok 5
not ok 6 # TODO
ok 7 # TODO
END
+cat >> exp <<END
+SKIP: b.test 1 # SKIP
+XPASS: b.test 2 # TODO
+XFAIL: b.test 3 # TODO
+ERROR: b.test - Bail out!${tab} ${tab}We're out of disk space.
+END
+
+# skip += 1, xpass += 1, xfail += 1, error += 1
+
+#------------------------------------------------------------------
+
# Bail out before the test plan.
-# pass += 2, error += 1
cat > c.test <<END
ok 1
ok 2
-Bail out! ... BOOOH!${tab}${sp}
+Bail out!BOOOH!${tab}${sp}
1..2
END
+cat >> exp <<END
+PASS: c.test 1
+PASS: c.test 2
+ERROR: c.test - Bail out!BOOOH!${tab}${sp}
+END
+
+# pass += 2, error += 1
+
+#------------------------------------------------------------------
+
# TAP input comprised only of a bailout directive.
-# error += 1
cat > d.test <<END
-Bail out!${tab}${sp}
+Bail out!
END
-# Bailout on the first line.
+echo "ERROR: d.test - Bail out!" >> exp
+
# error += 1
+
+#------------------------------------------------------------------
+
+# Bailout on the first line.
cat > e.test <<END
Bail out! mktemp -d: Permission denied
ok 1
END
+echo 'ERROR: e.test - Bail out! mktemp -d: Permission denied' >> exp
+
+# error += 1
+
+#------------------------------------------------------------------
+
# Doing the sums above, we have:
test_counts='total=11 pass=3 fail=1 xpass=1 xfail=1 skip=1 error=5'
@@ -93,26 +135,14 @@ cat stdout
count_test_results $test_counts
-grep '^PASS: a\.test 1$' stdout
-grep '^FAIL: a\.test 2$' stdout
-grep '^ERROR: a\.test - Bail out!$' stdout
-test `$FGREP -c ': a.test' stdout` -eq 3
-
-grep '^SKIP: b\.test 1$' stdout
-grep '^XPASS: b\.test 2$' stdout
-grep '^XFAIL: b\.test 3$' stdout
-grep '^ERROR: b\.test - Bail out! We'\''re out of disk space\.$' stdout
-test `$FGREP -c ': b.test' stdout` -eq 4
-
-grep '^PASS: c\.test 1$' stdout
-grep '^PASS: c\.test 2$' stdout
-grep '^ERROR: c\.test - Bail out! \.\.\. BOOOH!'"${tab}${sp}$" stdout
-test `$FGREP -c ': c.test' stdout` -eq 3
+LC_ALL=C sort exp > t
+mv -f t exp
-grep '^ERROR: d\.test - Bail out!$' stdout
-test `$FGREP -c ': d.test' stdout` -eq 1
+# We need the sort below to account for parallel make usage.
+grep ': [abcde]\.test' stdout | LC_ALL=C sort > got
-grep '^ERROR: e\.test - Bail out! mktemp -d: Permission denied$' stdout
-test `$FGREP -c ': e.test' stdout` -eq 1
+cat exp
+cat got
+diff exp got
:
diff --git a/tests/tap-basic.test b/tests/tap-basic.test
index 2ee8786..10b867d 100755
--- a/tests/tap-basic.test
+++ b/tests/tap-basic.test
@@ -15,14 +15,14 @@
# along with this program. If not, see <http://www.gnu.org/licenses/>.
# Basic TAP support:
-# - LOG_COMPILER support
-# - basic support for TODO and SKIP directives
-# - testsuite progress output on console
-# - runtime override of TESTS and TEST_LOGS
-# - correct counts of test results (both in summary and in
-# progress output on console)
+# - LOG_COMPILER support;
+# - basic support for TODO and SKIP directives, and "Bail out!" magic;
+# - testsuite progress output on console;
+# - runtime override of TESTS and TEST_LOGS;
+# - correct counts of test results (both in summary and in progress
+# output on console).
# Note that some of the features checked here are checked in other
-# test cases too, usually in more detail.
+# test cases too, usually in a more thorough and detailed way.
parallel_tests=yes
. ./defs || Exit 1
@@ -41,21 +41,38 @@ TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
## tap-driver script.
TEST_LOG_COMPILER = cat
TESTS = all.test
-END
-metacharacters=\''"\$!&()[]<>#;^?*'
+ok.test:
+ echo '1..3' > address@hidden
+ echo 'ok 1' >> address@hidden
+ echo 'not ok 2 # TODO' >>address@hidden
+ echo 'ok 3 # SKIP' >>address@hidden
+ cat address@hidden ;: For debugging.
+ mv -f address@hidden $@
+END
-cat > all.test <<END
-1..23
+cat > all.test << 'END'
+1..20
ok 1
-ok 2 foo
-ok 3 - foofoo
-ok 4 bar baz
-not ok 5 xfoo
-not ok 6 - xfoo
-not ok 7 xbar xbaz
-not ok 8 xfoo
-ok 5 $metacharacters
+ok 2 two
+ok 3 - three
+ok 4 four four
+not ok 5
+not ok 6 six
+not ok 7 - seven
+not ok 8 eight eight
+ok 9 # TODO
+ok 10 ten # TODO
+ok 11 - eleven # TODO
+ok 12 twelve twelve # TODO
+not ok 13 # TODO
+not ok 14 fourteen # TODO
+not ok 15 - fifteen # TODO
+not ok 16 sixteen sixteen # TODO
+ok 17 # SKIP
+ok 18 eighteen # SKIP
+ok 19 - nineteen # SKIP
+ok 20 twenty twenty # SKIP
END
$ACLOCAL
@@ -64,11 +81,96 @@ $AUTOMAKE
./configure
+# Basilar usage and testsuite progress output.
+
$MAKE check >stdout && { cat stdout; Exit 1; }
cat stdout
-count_test_results total=10 pass=2 fail=2 xpass=2 xfail=2 skip=2 error=0
+count_test_results total=20 pass=4 fail=4 xpass=4 xfail=4 skip=4 error=0
+
+test -f all.log
+test -f test-suite.log
+
+cat > exp << 'END'
+PASS: all.test 1
+PASS: all.test 2 two
+PASS: all.test 3 - three
+PASS: all.test 4 four four
+FAIL: all.test 5
+FAIL: all.test 6 six
+FAIL: all.test 7 - seven
+FAIL: all.test 8 eight eight
+XPASS: all.test 9 # TODO
+XPASS: all.test 10 ten # TODO
+XPASS: all.test 11 - eleven # TODO
+XPASS: all.test 12 twelve twelve # TODO
+XFAIL: all.test 13 # TODO
+XFAIL: all.test 14 fourteen # TODO
+XFAIL: all.test 15 - fifteen # TODO
+XFAIL: all.test 16 sixteen sixteen # TODO
+SKIP: all.test 17 # SKIP
+SKIP: all.test 18 eighteen # SKIP
+SKIP: all.test 19 - nineteen # SKIP
+SKIP: all.test 20 twenty twenty # SKIP
+END
+
+$FGREP ': all.test' stdout > got
+
+cat exp
+cat got
+diff exp got
+
+# Override TESTS from the command line.
+
+rm -f *.log *.test
+
+cat > bail.test <<'END'
+1..1
+Bail out!
+END
+
+TESTS=bail.test $MAKE -e check >stdout && { cat stdout; Exit 1; }
+cat stdout
+
+count_test_results total=1 pass=0 fail=0 xpass=0 xfail=0 skip=0 error=1
+
+test ! -f all.log
+test -f bail.log
+test -f test-suite.log
+
+grep '^ERROR: bail.test - Bail out!' stdout
+$FGREP bail.test test-suite.log
+test `$FGREP ': bail.test' stdout` -eq 1
+$FGREP all.test stdout test-suite.log && Exit 1
+
+# Override TEST_LOGS from the command line, making it point to a test
+# (ok.test) that have to be generated at make time.
+
+rm -f *.log *.test
+
+TEST_LOGS=ok.log $MAKE -e check >stdout || { cat stdout; Exit 1; }
+cat stdout
+
+count_test_results total=3 pass=1 fail=0 xpass=0 xfail=1 skip=1 error=0
+
+test -f ok.test
+test -f ok.log
+test ! -f all.log
+test ! -f bail.log
+test -f test-suite.log
+
+$EGREP '(bail|all)\.test' stdout test-suite.log && Exit 1
+
+cat > exp << 'END'
+PASS: ok.test 1
+XFAIL: ok.test 2
+SKIP: ok.test 3
+END
+
+$FGREP ': ok.test' stdout > got
-Exit 1; # TODO ALL
+cat exp
+cat got
+diff exp got
:
diff --git a/tests/tap-color.test b/tests/tap-color.test
index f4a5f21..05a01ac 100755
--- a/tests/tap-color.test
+++ b/tests/tap-color.test
@@ -34,7 +34,6 @@ blu="$esc\[1;34m"
mgn="$esc\[0;35m"
std="$esc\[m"
-
# Check that grep can parse nonprinting characters.
# BSD 'grep' works from a pipe, but not a seekable file.
# GNU or BSD 'grep -a' works on files, but is not portable.
@@ -60,14 +59,14 @@ cat > all.test << 'END'
1..5
ok 1 - foo
# Hi! I shouldn't be colorized!
-not ok 2 - bar # TODO difficult
-ok 3 - baz # SKIP I hate baz
+not ok 2 - bar # TODO td
+ok 3 - baz # SKIP sk
not ok 4 - quux
ok 5 - zardoz # TODO
END
cat > skip.test << 'END'
-1..0 # skip whole script
+1..0 # Whole script not run
END
cat > errors.test << 'END'
@@ -76,9 +75,6 @@ ok 1
Bail out!
END
-# Remember to test also test that get completely skipped:
-# plan: 1..0
-
$ACLOCAL
$AUTOCONF
$AUTOMAKE
@@ -92,15 +88,15 @@ test_color ()
{
# Not a useless use of cat; see above comments about grep.
cat stdout | grep "^${grn}PASS${std}: all\.test 1 - foo$"
- cat stdout | grep "^${lgn}XFAIL${std}: all\.test 2 - bar (difficult)$"
- cat stdout | grep "^${blu}SKIP${std}: all\.test 3 - baz (I hate baz)$"
+ cat stdout | grep "^${lgn}XFAIL${std}: all\.test 2 - bar # TODO td$"
+ cat stdout | grep "^${blu}SKIP${std}: all\.test 3 - baz # SKIP sk$"
cat stdout | grep "^${red}FAIL${std}: all\.test 4 - quux$"
cat stdout | grep "^${red}XPASS${std}: all\.test 5 - zardoz$"
- cat stdout | grep "^${blu}SKIP${std}: skip\.test (skip whole script)$"
+ cat stdout | grep "^${blu}SKIP${std}: skip\.test # Whole script not run$"
cat stdout | grep "^${grn}PASS${std}: bailout\.test 1$"
cat stdout | grep "^${mgn}ERROR${std}: all\.test - Bail out!$"
- # Diagnostic messages shouldn't be colored.
- cat stdout | grep "^DIAG: all\.test # Hi! I shouldn't be colorized!$"
+ # Diagnostic messages shouldn't be colorized.
+ cat stdout | grep "^# all\.test: Hi! I shouldn't be colorized!$"
:
}
@@ -124,7 +120,7 @@ test_no_color ()
cat stdout | grep "XFAIL.*:"
cat stdout | grep "XPASS.*:"
cat stdout | grep "ERROR.*:"
- cat stdout | grep "DIAG.*:"
+ cat stdout | grep "^#"
cat stdout | grep 'test.*expected'
cat stdout | grep 'test.*not run'
cat stdout | grep '===='
diff --git a/tests/tap-no-ok-skip.test b/tests/tap-deps.test
similarity index 51%
copy from tests/tap-no-ok-skip.test
copy to tests/tap-deps.test
index 57b8d7a..a190de0 100755
--- a/tests/tap-no-ok-skip.test
+++ b/tests/tap-deps.test
@@ -14,9 +14,8 @@
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
-# TAP support:
-# - a "not ok # SKIP" line should count as a failure, for consistency
-# with the prove(1) utility.
+# Basic TAP test protocol support:
+# - dependencies between test scripts
parallel_tests=yes
. ./defs || Exit 1
@@ -31,27 +30,67 @@ END
cat > Makefile.am << 'END'
TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
-TEST_LOG_COMPILER = cat
-TESTS = all.test
+## The tests are deliberately listed in inversed order here.
+TESTS = c.test b.test a.test
+c.log: a.log
+b.log: c.log
+EXTRA_DIST = $(TESTS)
END
-cat > all.test <<'END'
-1..4
-not ok # SKIP
-not ok 2 # SKIP
-not ok - foo # SKIP
-not ok 4 - bar # SKIP
+cat > a.test << 'END'
+echo 1..2
+echo ok 1
+sleep '3' # FIXME: quotes to please maintainer-check
+echo ok 2
+: > a.run
END
+cat > b.test << 'END'
+echo 1..2
+if test -f a.run; then
+ echo not ok 1
+else
+ echo ok 1
+fi
+sleep '3' # FIXME: quotes to please maintainer-check
+echo ok 2
+: > b.run
+END
+
+cat > c.test << 'END'
+echo 1..1
+test -f b.run || { echo 'Bail out!'; exit 1; }
+echo ok 1
+rm -f a.run b.run
+END
+
+
$ACLOCAL
$AUTOCONF
$AUTOMAKE
./configure
-$MAKE check >stdout && { cat stdout; Exit 1; }
+$MAKE check >stdout || { cat stdout; Exit 1; }
cat stdout
-count_test_results total=4 pass=0 fail=4 skip=0 xpass=0 xfail=0 error=0
+count_test_results total=5 pass=3 fail=0 xpass=0 xfail=1 skip=1 error=0
+
+cat > exp << 'END'
+PASS: a.test 1
+PASS: a.test 2
+PASS: b.test 1
+PASS: b.test 2
+PASS: c.test 1
+END
+
+grep ': [abc]\.test' stdout > got
+
+cat exp
+cat got
+diff exp got
+
+# TODO: it would be nice to redo the checks also forcing the use of
+# parallel make
:
diff --git a/tests/tap-diagnostic.test b/tests/tap-diagnostic.test
index 9988f1e..54fcb47 100755
--- a/tests/tap-diagnostic.test
+++ b/tests/tap-diagnostic.test
@@ -51,7 +51,7 @@ ok 3 - bar # SKIP
# ${tab}${tab} ${tab}More leading whitespace is kept.
# Trailing whitespace should be kept, 1.$sp
# Trailing whitespace should be kept, 2.$tab
-# All${tab}${tab}whitespace should ${tab} be kept. $tab
+# All${tab}${tab}whitespace should ${tab} be kept. $tab
# Shell metacharacters here: $metacharacters
ok - zardoz
# Following lines shouldn't be parsed as test results.
@@ -73,31 +73,30 @@ ${tab}# Leading whitespace before "#", not a TAP diagnostic
line.
END
cat > exp <<END
-DIAG: all.test # Hi! I'm a comment.
-DIAG: all.test # Tests begin.
+# all.test: Hi! I'm a comment.
+# all.test: Tests begin.
PASS: all.test 1
XFAIL: all.test 2 - foo
SKIP: all.test 3 - bar
-DIAG: all.test # Tests end.
-DIAG: all.test #No leading whitespace is OK.
-DIAG: all.test # ${tab}${tab} ${tab}More leading whitespace is kept.
-DIAG: all.test # Leading whitespace should be stripped, 3.
-DIAG: all.test # Trailing whitespace should be kept, 1.$sp
-DIAG: all.test # Trailing whitespace should be kept, 2.$tab
-DIAG: all.test # All${tab}${tab}whitespace should ${tab} be kept. $tab
-DIAG: all.test # Shell metacharacters here: $metacharacters
+# all.test Tests end.
+# all.test:No leading whitespace is OK.
+# all.test: ${tab}${tab} ${tab}More leading whitespace is kept.
+# all.test: Trailing whitespace should be kept, 1.$sp
+# all.test: Trailing whitespace should be kept, 2.$tab
+# all.test: All${tab}${tab}whitespace should ${tab} be kept. $tab
+# all.test # Shell metacharacters here: $metacharacters
PASS: all.test - 4 - zardoz
-DIAG: all.test # Following lines shouldn't be parsed as test results.
-DIAG: all.test # ok
-DIAG: all.test #ok
-DIAG: all.test # not ok
-DIAG: all.test #not ok
-DIAG: all.test # Bail out!
-DIAG: all.test #Bail out!
-DIAG: all.test # SKIP
-DIAG: all.test #SKIP
-DIAG: all.test # TODO
-DIAG: all.test #TODO
+# all.test: Following lines shouldn't be parsed as test results.
+# all.test: ok
+# all.test:ok
+# all.test: not ok
+# all.test:not ok
+# all.test: Bail out!
+# all.test:Bail out!
+# all.test: SKIP
+# all.test:SKIP
+# all.test: TODO
+# all.test:TODO
END
$ACLOCAL
@@ -108,13 +107,13 @@ $AUTOMAKE
$MAKE check >stdout || { cat stdout; Exit 1; }
cat stdout
-grep '^DIAG:' stdout && Exit 1
+$EGREP '(^#|#.*all\.test)' stdout && Exit 1
count_test_results total=3 pass=3 fail=0 xpass=0 xfail=1 skip=1 error=0
echo 'AM_TEST_LOG_DRIVER_FLAGS = --comments' >> Makefile
$MAKE check >stdout || { cat stdout; Exit 1; }
cat stdout
-LC_ALL=C grep '^[A-Z][A-Z]*:' stdout > got
+LC_ALL=C $EGREP "(^[ $tab]*#|all\.test)" stdout > got
cat exp
cat got
diff exp got
@@ -123,7 +122,7 @@ count_test_results total=3 pass=3 fail=0 xpass=0 xfail=1
skip=1 error=0
TEST_LOG_DRIVER_FLAGS="--no-comments" $MAKE -e check >stdout \
|| { cat stdout; Exit 1; }
cat stdout
-grep '^DIAG:' stdout && Exit 1
+$EGREP '(^#|#.*all\.test)' stdout && Exit 1
count_test_results total=3 pass=3 fail=0 xpass=0 xfail=1 skip=1 error=0
:
diff --git a/tests/tap-fancy.test b/tests/tap-fancy.test
index 96403a5..762e315 100755
--- a/tests/tap-fancy.test
+++ b/tests/tap-fancy.test
@@ -34,6 +34,12 @@ TEST_LOG_COMPILER = cat
TESTS = all.test
END
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
#
# From manpage Test::Harness::TAP(3):
#
@@ -48,7 +54,7 @@ END
#
cat > all.test <<END
-1..19
+1..21
ok? a question
not ok? a question
@@ -71,24 +77,73 @@ not ok ${tab}${tab} 12
ok# SKIP
ok${tab}#SKIP--who cares?
ok?#SKIP!
+ok!#SKIP?
not ok# TODO
not ok${tab}#TODO--who cares?
-ok?#TODO!
+not ok?#TODO!
+not ok!#TODO?
-ok~~#TODO~~
+ok~#TODO
END
+$MAKE check >stdout && { cat stdout; Exit 1; }
+cat stdout
-$ACLOCAL
-$AUTOCONF
-$AUTOMAKE
+count_test_results total=21 pass=6 fail=6 xfail=4 xpass=1 skip=4 error=0
-./configure
+#
+# "Weird" characters support.
+#
+
+weirdchars=\''"$!&()[]<>#;^?*/@%=,.:'
+
+cat > all.test <<END
+1..6
+ok $weirdchars
+not ok $weirdchars
+ok $weirdchars # TODO
+not ok $weirdchars # TODO
+ok $weirdchars # SKIP
+Bail out! $weirdchars
+END
+
+$MAKE check >stdout && { cat stdout; Exit 1; }
+cat stdout
+
+count_test_results total=6 pass=1 fail=1 xfail=1 xpass=1 skip=1 error=1
+$FGREP "PASS: all.test 1 $weirdchars" stdout
+$FGREP "FAIL: all.test 2 $weirdchars" stdout
+$FGREP "XPASS: all.test 3 $weirdchars" stdout
+$FGREP "XPASS: all.test 4 $weirdchars" stdout
+$FGREP "SKIP: all.test 5 $weirdchars" stdout
+$FGREP "PASS: all.test - Bail out! $weirdchars" stdout
+
+#
+# Trailing backslashes does not confuse the parser.
+#
+
+bs='\'
+
+cat > all.test <<END
+1..6
+ok $bs
+not ok $bs
+ok $bs
+not ok $bs
+ok $bs
+Bail out! $bs
+END
$MAKE check >stdout && { cat stdout; Exit 1; }
cat stdout
-count_test_results total=19 pass=6 fail=6 xfail=3 xpass=1 skip=3
+count_test_results total=6 pass=1 fail=1 xfail=1 xpass=1 skip=1 error=1
+grep '^PASS: all\.test 1 \\$' stdout
+grep '^FAIL: all\.test 2 \\$' stdout
+grep '^XPASS: all\.test 3 \\$' stdout
+grep '^XPASS: all\.test 4 \\$' stdout
+grep '^SKIP: all\.test 5 \\$' stdout
+grep '^PASS: all\.test - Bail out! \\$' stdout
:
diff --git a/tests/tap-fancy2.test b/tests/tap-fancy2.test
index ae5e0ef..9cd2aa7 100755
--- a/tests/tap-fancy2.test
+++ b/tests/tap-fancy2.test
@@ -112,13 +112,13 @@ skip=125 # = 4 * 32 - 3
error=0
total=`expr $pass + $fail + $xfail + $xpass + $skip`
-# Even nastier!
+# Even nastier! But accordingly to the specifics, it should still work.
for result in 'ok' 'not ok'; do
echo "${result}{[(<#${tab}TODO>)]}" >> all.test
done
echo "ok{[(<#${tab}SKIP>)]}" >> all.test
-# We have to update the some test counts.
+# We have to update some test counts.
xfail=`expr $xfail + 1`
xpass=`expr $xpass + 1`
skip=`expr $skips + 1`
diff --git a/tests/tap-log.test b/tests/tap-log.test
index 7bccd3c..b3a3e0a 100755
--- a/tests/tap-log.test
+++ b/tests/tap-log.test
@@ -17,6 +17,7 @@
# TAP support:
# - log file creation
# - log file removal
+# - stdout and stderr of a TAP-generating test script go in its log file
# - TEST_SUITE_LOG redefinition, at either automake or make time
# - VERBOSE environment variable support
@@ -33,7 +34,6 @@ END
cat > Makefile.am << 'END'
TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
-TEST_LOG_COMPILER = cat
TESTS = pass.test skip.test xfail.test fail.test xpass.test error.test
TEST_SUITE_LOG = global.log
END
@@ -41,40 +41,49 @@ END
marker=%%% # Custom marker, for use in grepping checks.
cat > pass.test <<END
-1..1
-# $marker pass $marker
-ok 1
+#! /bin/sh -e
+echo 1..1
+echo "# $marker pass $marker" >&2
+echo "ok 1"
END
cat > skip.test <<END
-1..1
-# $marker skip $marker
-not ok 1 # SKIP
+#! /bin/sh -e
+echo 1..1
+echo "# $marker skip $marker"
+echo "ok 1 # SKIP"
END
cat > xfail.test <<END
-1..1
-# $marker xfail $marker
-not ok 1 # TODO
+#! /bin/sh -e
+echo 1..1
+echo "# $marker xfail $marker" >&2
+echo "not ok 1 # TODO"
END
cat > fail.test <<END
-1..1
-# $marker fail $marker
-not ok 1
+#! /bin/sh -e
+echo 1..1
+echo "# $marker fail $marker"
+echo "not ok 1"
END
cat > pass.test <<END
-1..1
-# $marker xpass $marker
+#! /bin/sh -e
+echo 1..1
+echo "# $marker xpass $marker" >&2
ok 1 # TODO
END
cat > error.test <<END
-1..1
-Bail out!
+#! /bin/sh -e
+echo 1..1
+echo "# $marker error $marker"
+echo 'Bail out!'
END
+chmod a+x *.test
+
$ACLOCAL
$AUTOCONF
$AUTOMAKE
diff --git a/tests/tap-merge-stdout-stderr.test
b/tests/tap-merge-stdout-stderr.test
index ef50818..d02f412 100755
--- a/tests/tap-merge-stdout-stderr.test
+++ b/tests/tap-merge-stdout-stderr.test
@@ -16,8 +16,7 @@
# TAP support:
# - The Automake TAP driver has an option that instruct it to read TAP
-# input only from the stdout of the test command, not also its stderr
-# (as is required by the TAP specifications).
+# input also from the stderr of the test command, not only its stdout.
parallel_tests=yes
. ./defs || Exit 1
@@ -32,6 +31,7 @@ END
cat > Makefile.am << 'END'
TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+AM_TEST_LOG_DRIVER_FLAGS = --comments --merge
TESTS = all.test
END
@@ -47,7 +47,7 @@ echo 1..4
echo ok 1 >&2
echo ok 2
echo "not ok 3 # TODO" >&2
-echo "not ok 4 # SKIP"
+echo "ok 4 # SKIP"
echo "# foo foo foo" >&2
END
@@ -55,7 +55,7 @@ $MAKE check >stdout || { cat stdout; Exit 1; }
cat stdout
count_test_results total=4 pass=2 fail=0 xpass=0 xfail=1 skip=1 error=0
-grep '^DIAG:.*foo foo foo' stdout
+grep '^# all\.test: foo foo foo' stdout
cat > all.test <<END
#!/bin/sh
@@ -69,4 +69,19 @@ cat stdout
count_test_results total=2 pass=1 fail=0 xpass=0 xfail=0 skip=0 error=1
+# See that the option `--no-merge' can override the effect of `--merge'.
+
+# Sanity check: both the option are passed to tap-driver.
+TEST_LOG_DRIVER_FLAGS=--no-merge $MAKE -e -n check >output 2>&1 \
+ || { cat output; Exit 1; }
+grep 'tap-driver.*--merge .*--no-merge' output
+
+TEST_LOG_DRIVER_FLAGS=--no-merge $MAKE -e check >stdout \
+ || { cat stdout; Exit 1; }
+cat stdout
+
+count_test_results total=1 pass=1 fail=0 xpass=0 xfail=0 skip=0 error=0
+
+
+
:
diff --git a/tests/tap-more.test b/tests/tap-more.test
index 345d7e1..c19504b 100755
--- a/tests/tap-more.test
+++ b/tests/tap-more.test
@@ -19,6 +19,7 @@
# - VPATH builds
# - use with parallel make (if supported)
# - basic use of diagnostic messages (lines beginning with "#")
+# - flags for TAP driver defined through AC_SUBST in configure.ac
# - messages generated by the testsuite harness reference the
# correct test script(s)
# - "make distcheck" works
@@ -31,6 +32,7 @@ cp "$top_testsrcdir"/lib/tap-driver . \
cat >> configure.in <<END
AC_SUBST([PERL], ['$PERL'])
+AC_SUBST([AM_TEST_LOG_DRIVER_FLAGS], ['--comments'])
AC_OUTPUT
END
@@ -43,11 +45,11 @@ END
cat > 1.test <<'END'
#! /bin/sh
echo 1..2
-echo ok 1 mu
+echo ok 1 - mu
if test -f not-skip; then
echo "not ok 2 zardoz"
else
- echo "not ok 2 zardoz # SKIP"
+ echo "ok 2 zardoz # SKIP"
fi
END
@@ -118,15 +120,15 @@ for try in 0 1; do
cat stdout
count_test_results total=6 pass=4 fail=0 xpass=0 xfail=1 skip=1 error=0
grep '^PASS: 1\.test 1 - mu$' stdout
- grep '^SKIP: 1\.test 2 - zardoz$' stdout
- test `$FGREP -c : 1.test stdout` -eq 2
+ grep '^SKIP: 1\.test 2 zardoz # SKIP$' stdout
+ test `$FGREP -c ': 1.test' stdout` -eq 2
grep '^PASS: 2\.test 1$' stdout
- grep '^XFAIL: 2\.test 2 (not implemented)$' stdout
+ grep '^XFAIL: 2\.test 2 # TODO not implemented$' stdout
grep '^PASS: 2\.test 3$' stdout
- test `$FGREP -c : 2.test stdout` -eq 3
+ test `$FGREP -c ': 2.test' stdout` -eq 3
grep '^PASS: 3\.test 1 - blah blah blah$' stdout
- grep '^DIAG: 3\.test # Some diagnostic$' stdout
- test `$FGREP -c : 3.test stdout` -eq 2
+ grep '^# 3\.test: Some diagnostic$' stdout
+ test `$FGREP -c ': 3.test' stdout` -eq 2
# Failure.
: > not-skip
@@ -135,16 +137,16 @@ for try in 0 1; do
cat stdout
count_test_results total=7 pass=3 fail=1 xpass=0 xfail=1 skip=0 error=1
grep '^PASS: 1\.test 1 - mu$' stdout
- grep '^FAIL: 1\.test 2 - zardoz$' stdout
- test `$FGREP -c : 1.test stdout` -eq 2
+ grep '^FAIL: 1\.test 2 zardoz$' stdout
+ test `$FGREP -c ': 1.test' stdout` -eq 2
grep '^PASS: 2\.test 1$' stdout
- grep '^XFAIL: 2\.test 2 (not implemented)$' stdout
+ grep '^XFAIL: 2\.test 2 # TODO not implemented$' stdout
grep '^PASS: 2\.test 3$' stdout
- test `$FGREP -c : 2.test stdout` -eq 3
+ test `$FGREP -c ': 2.test' stdout` -eq 3
grep '^PASS: 3\.test 1 - blah blah blah$' stdout
- grep '^DIAG: 3\.test # Some diagnostic$' stdout
+ grep '^# 3\.test: Some diagnostic$' stdout
grep '^ERROR: 3\.test - Bail out! Kernel Panic$' stdout
- test `$FGREP -c : 3.test stdout` -eq 3
+ test `$FGREP -c ': 3.test' stdout` -eq 3
cd $srcdir
diff --git a/tests/tap-more2.test b/tests/tap-more2.test
new file mode 100755
index 0000000..870495c
--- /dev/null
+++ b/tests/tap-more2.test
@@ -0,0 +1,90 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# More on TAP support:
+# - more LOG_COMPILER at once for TAP tests
+# - binary programs in $(TESTS)
+# - interactions with `check_*' variables
+
+required='cc native'
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_PROG_CC
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_EXTENSIONS = .sh .tap
+
+tap_driver = $(PERL) $(srcdir)/tap-driver
+
+LOG_DRIVER = $(tap_driver)
+SH_LOG_DRIVER = $(tap_driver)
+TAP_LOG_DRIVER = $(tap_driver)
+
+DATA_LOG_COMPILER = cat
+SH_LOG_COMPILER = $(SHELL)
+LOG_COMPILER =
+
+EXTRA_DIST = baz.tap
+
+check_SCRIPTS = bar.sh
+bar.sh: Makefile
+ echo '#!/bin/sh' > address@hidden
+ echo 'echo 1..1' >> address@hidden
+ echo 'echo "not ok 1 # TODO"' >> address@hidden
+ chmod a-w address@hidden && mv -f address@hidden $@
+
+check_PROGRAMS = foo-test
+foo_test_SOURCES = foo.c
+
+TESTS = foo-test $(check_SCRIPTS) baz.tap
+END
+
+cat > foo.c <<'END'
+#include <stdio.h>
+int maint (void)
+{
+ printf ("1..1\n");
+ printf ("ok 1\n");
+ return 0;
+}
+END
+
+cat > baz.tap << 'END'
+1..1
+ok 1 # SKIP
+END
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+for target in check distcheck; do
+ $MAKE $target >stdout || { cat stdout; Exit 1; }
+ cat stdout
+ count_test_results total=3 pass=1 fail=0 xpass=0 xfail=1 skip=1 error=0
+done
+
+:
diff --git a/tests/tap-no-merge-stdout-stderr.test
b/tests/tap-no-merge-stdout-stderr.test
index 8083c93..30dc460 100755
--- a/tests/tap-no-merge-stdout-stderr.test
+++ b/tests/tap-no-merge-stdout-stderr.test
@@ -15,8 +15,8 @@
# along with this program. If not, see <http://www.gnu.org/licenses/>.
# TAP support:
-# - By default, TAP input is read from both the stderr and stdout of the
-# test command (similarly with what happens with "prove --merge").
+# - By default, TAP input is only from the stdout (and not the stderr)
+# of the test command.
parallel_tests=yes
. ./defs || Exit 1
@@ -44,9 +44,10 @@ cat > all.test <<END
#!/bin/sh
echo 1..2
echo 'not ok' >&2
+echo 'not ok 1 # TODO' >&2
echo 'ok 1'
echo '# foo foo foo' >&2
-echo 'ok 2 # TODO' >&2
+echo 'ok 44 # TODO' >&2
echo 'Bail out!' >&2
echo 'ok 2 # SKIP'
END
@@ -55,6 +56,6 @@ $MAKE check >stdout || { cat stdout; Exit 1; }
cat stdout
count_test_results total=2 pass=1 fail=0 xpass=0 xfail=0 skip=1 error=0
-$EGREP '(DIAG:|foo foo foo)' stdout && Exit 1
+$EGREP '^#|foo foo foo' stdout && Exit 1
:
diff --git a/tests/tap-no-spurious.test b/tests/tap-no-spurious.test
index ba1586b..2f99ad2 100755
--- a/tests/tap-no-spurious.test
+++ b/tests/tap-no-spurious.test
@@ -42,37 +42,46 @@ echo 1..3 > all.test
cat >> all.test <<END
ok 1 # XFAIL
ok 2 # SKIPPED
-ok 3 # TODOALL
+ok 3 # TO DO
+ok 4 # TODOALL
+ok 5 # FIXME
END
-# According to documentation of Test::Harness::TAP(3)
+# According to documentation of Test::Harness::TAP(3):
+#
# ``Lines written to standard output matching /^(not )?ok\b/
# must be interpreted as test lines. All other lines must
# not be considered test output.''
cat >> all.test <<END
ok
- ok 4
+ ok 1
${tab}ok
-${tab}ok 6
-ok7
+${tab}ok 1
+ok1
+ok23
not ok
- not ok 9
+ not ok 1
${tab}not ok
-${tab}not ok 11
-not ok12
+${tab}not ok 1
+not ok1
+not ok23
notok
-notok 14
+notok 1
not${tab}ok
-not${tab}ok 16
+not${tab}ok 1
not ok
-not ok 18
+not ok 1
no ok
-no ok 20
+no ok 1
# ok
# not ok
-# ok 23
-# not ok 24
+# ok 1
+# not ok 1
+#ok
+#not ok
+#ok 1
+#not ok 1
END
# The prove(1) utility doesn't bail out on these, so our driver
@@ -100,6 +109,6 @@ $AUTOMAKE
$MAKE check >stdout && { cat stdout; Exit 1; }
cat stdout
-count_test_results total=3 pass=3 fail=0 xpass=0 xfail=0 skip=0 error=0
+count_test_results total=5 pass=5 fail=0 xpass=0 xfail=0 skip=0 error=0
:
diff --git a/tests/tap-no-ok-skip.test b/tests/tap-not-ok-skip.test
similarity index 100%
rename from tests/tap-no-ok-skip.test
rename to tests/tap-not-ok-skip.test
diff --git a/tests/tap-numeric-description.test
b/tests/tap-numeric-description.test
index 5275b77..5238a70 100755
--- a/tests/tap-numeric-description.test
+++ b/tests/tap-numeric-description.test
@@ -40,8 +40,8 @@ set 5 7 45 11 7 3 6 9 1000 1
cat > all.test <<END
1..10
-ok - ${1}
-ok 2 ${2}
+ok 1 ${1}
+ok - ${2}
not ok 3 ${3}
not ok - ${4}
ok 5 ${5} # SKIP
@@ -49,10 +49,21 @@ ok - ${6} # SKIP
not ok 7 ${7} # TODO
not ok - ${8} # TODO
ok 9 ${9} # TODO
-ok - 10 ${10} # TODO
+ok - ${10} # TODO
END
-chmod a+x all.test
+cat > exp <<END
+PASS 1 ${1}
+PASS 2 - ${2}
+FAIL 3 ${3}
+FAIL 4 - ${4}
+SKIP 5 ${5} # SKIP
+SKIP 6 - ${6} # SKIP
+XFAIL 7 ${7} # TODO
+XFAIL 8 - ${8} # TODO
+XPASS 9 ${9} # TODO
+XPASS 10 - ${10} # TODO
+END
$ACLOCAL
$AUTOCONF
@@ -65,20 +76,14 @@ cat stdout
count_test_results total=10 pass=2 fail=2 xpass=2 xfail=2 skip=2 error=0
-# Usage: do grep RESULT NUM DESC
-do_grep () { grep "^$1: all\\.test $2 - $3$" stdout; }
-
-do_grep PASS 1 $1
-do_grep PASS 2 $2
-do_grep FAIL 3 $3
-do_grep FAIL 4 $4
-do_grep SKIP 5 $5
-do_grep SKIP 6 $6
-do_grep XPASS 7 $7
-do_grep XPASS 8 $8
-do_grep XFAIL 9 $9
-do_grep XFAIL 10 ${10}
-
-$EGREP ": all\.test([^:]|.*\[|$)" stdout && Exit 1
+$FGREP ': all.test' stdout > got
+
+sed 's/^/all.test: /' exp > t
+mv -f t exp
+
+cat exp
+cat got
+diff exp got
+
:
diff --git a/tests/tap-out-of-order.test b/tests/tap-out-of-order.test
index df71983..b1821cd 100755
--- a/tests/tap-out-of-order.test
+++ b/tests/tap-out-of-order.test
@@ -62,6 +62,7 @@ END
# TAP::Parser documentation. Keep it in sync with the example
# there.
cat > d.test <<'END'
+1..5
ok 1
ok 2
ok 15
diff --git a/tests/tap-passthrough.test b/tests/tap-passthrough.test
index 21ad5ad..4ad0bbc 100755
--- a/tests/tap-passthrough.test
+++ b/tests/tap-passthrough.test
@@ -56,14 +56,15 @@ ok 2
ok - foo
ok 4 - x
This is not a TAP line, but should still be copied in the log file!
+# diagnostic
not ok # TODO low priority
-not ok # SKIP who cares?
+ok # SKIP who cares?
This is not in test-suite.log, since ok.test does not fail.
$weirdchars
END
st=0
-TESTSL=ok.test $MAKE -e check || st=$?
+TESTS=ok.test $MAKE -e check || st=$?
cat ok.log
cat test-suite.log
test $st -eq 0 || Exit 1
@@ -72,20 +73,21 @@ for rx in \
'1\.\.6' \
'TAP plan in the previous line\.' \
"ok${tab}" \
- 'ok 2' \
- 'ok - foo' \
- 'ok 4 - x' \
- 'This is not a TAP line, but should still be copied in the log file!' \
- 'not ok # TODO low priority' \
- 'not ok # SKIP who cares?' \
- 'This is not in test-suite\.log, since ok\.test does not fail\.' \
+ 'ok 2' \
+ 'ok - foo' \
+ 'ok 4 - x' \
+ ' This is not a TAP line, but should still be copied in the log file!' \
+ '# diagnostic' \
+ 'not ok # TODO low priority' \
+ 'ok # SKIP who cares?' \
+ 'This is not in test-suite\.log, since ok\.test does not fail\.' \
; do
grep "^$rx$" foo.log
done
$FGREP "$weirdchars" foo.log
$FGREP 'foo.test' test-suite.log && Exit 1
-$FGREP -i 'this is not in test-suite.log' test-suite.log && Exit 1
+$FGREP 'not in test-suite.log' test-suite.log && Exit 1
#
# Mixed failing/successful tests.
@@ -104,6 +106,7 @@ ok${tab}
ok 2
not ok - foo
not ok 4 - x
+# diagnostic ko
bar${tab}bar${tab}bar
ok # TODO dunno
$weirdchars
@@ -150,6 +153,7 @@ for rx in \
'ok 2' \
'not ok - foo' \
'not ok 4 - x' \
+ '# diagnostic ko' \
" bar${tab}bar${tab}bar" \
'ok # TODO dunno' \
; do
diff --git a/tests/tap-plan.test b/tests/tap-plan.test
index 6183204..b549b3f 100755
--- a/tests/tap-plan.test
+++ b/tests/tap-plan.test
@@ -33,7 +33,7 @@ END
cat > Makefile.am << 'END'
TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
TEST_LOG_COMPILER = cat
-TESTS = top.test bot.test none.test
+TESTS = top.test bot.test
END
$ACLOCAL
@@ -45,9 +45,9 @@ $AUTOMAKE
cat > top.test <<END
1..3
ok 1
-not ok 2 # SKIP
+ok 2 # SKIP
ok 3
-# a tailing comment
+# a trailing comment
END
cat > bot.test <<END
@@ -60,18 +60,11 @@ ok
1..4
END
-cat > none.test <<END
-# 1..3
-# the previous line shouldn't be picked up by mistake.
-ok 1
-ok 2
-END
-
# Check that the plans doesn't cause any problem
$MAKE check >stdout || { cat stdout; Exit 1; }
cat stdout
-count_test_results total=9 pass=7 xfail=1 skip=1 fail=0 xpass=0 error=0
+count_test_results total=7 pass=5 xfail=1 skip=1 fail=0 xpass=0 error=0
# Now check that the plans had been really considered.
@@ -80,7 +73,7 @@ mv -f t top.test
sed '$s/.*/1..9/' bot.test > t
mv -f t bot.test
-$MAKE check >stdout || { cat stdout; Exit 1; }
+$MAKE check >stdout && { cat stdout; Exit 1; }
cat stdout
count_test_results total=11 pass=7 xfail=1 skip=1 fail=0 xpass=0 error=2
grep '^ERROR: top\.test' stdout
diff --git a/tests/tap-realtime.test b/tests/tap-realtime.test
index ab26084..f5b9496 100755
--- a/tests/tap-realtime.test
+++ b/tests/tap-realtime.test
@@ -48,20 +48,14 @@ END
cat > all.test <<'END'
#! /bin/sh
-echo 1..9
+echo 1..3
echo ok 1
-echo ok 2
-echo ok 3
: > milestone-1
-sleep '20' # FIXME: quoting required to placate maintainer-check
-echo ok 4
-echo ok 5
-echo ok 6
+sleep '10' # FIXME: quoting required to placate maintainer-check
+echo ok 2
: > milestone-2
-sleep '20' # FIXME: quoting required to placate maintainer-check
-echo ok 7
-echo ok 8
-echo ok 9
+sleep '10' # FIXME: quoting required to placate maintainer-check
+echo ok 3
END
chmod a+x all.test
@@ -83,17 +77,20 @@ MAKE=$MAKE expect -f expect-make > stdout &
while test ! -f milestone-1; do sleep '1'; done
cat stdout
-test `$FGREP ': all.test' stdout` -eq 3
-for i in 1 2 3; do grep "^PASS: all\.test $i$"; done
+grep '^PASS: all\.test 1$' stdout
+test `$FGREP -c ': all.test'` -eq 1
while test ! -f milestone-2; do sleep '1'; done
cat stdout
-test `$FGREP ': all.test' stdout` -eq 6
-for i in 1 2 3 4 5 6; do grep "^PASS: all\.test $i$"; done
+grep '^PASS: all\.test 1$' stdout
+grep '^PASS: all\.test 2$' stdout
+test `$FGREP -c ': all.test'` -eq 2
wait
cat stdout
-test `$FGREP ': all.test' stdout` -eq 9
-for i in 1 2 3 4 5 6 7 8 9; do grep "^PASS: all\.test $i$"; done
+grep '^PASS: all\.test 1$' stdout
+grep '^PASS: all\.test 2$' stdout
+grep '^PASS: all\.test 3$' stdout
+test `$FGREP -c ': all.test'` -eq 2
:
diff --git a/tests/tap-recheck-logs.test b/tests/tap-recheck-logs.test
index 01ff155..30b9262 100755
--- a/tests/tap-recheck-logs.test
+++ b/tests/tap-recheck-logs.test
@@ -1,5 +1,5 @@
#! /bin/sh
-# Copyright (C) 2009, 2010, 2011 Free Software Foundation, Inc.
+# Copyright (C) 2011 Free Software Foundation, Inc.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
@@ -20,41 +20,35 @@
parallel_tests=yes
. ./defs || Exit 1
-Exit 1 # TODO
-
-cat >> configure.in << 'END'
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
AC_OUTPUT
END
cat > Makefile.am << 'END'
-TEST_SUITE_LOG = mylog.log
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TEST_LOG_COMPILER = cat
TESTS = foo.test bar.test baz.test
-XFAIL_TESTS = bar.test
-foo.log: bar.log
-bar.log: baz.log
+baz.log: zardoz
END
-# foo.test and bar.test sleep to ensure their logs are always strictly newer
-# than the logs of their prerequisites, for HP-UX make. The quoting pleases
-# maintainer-check.
-cat >>foo.test <<'END'
-#! /bin/sh
-echo "this is $0"
-sleep '1'
-exit 0
+: > zardoz
+
+cat > foo.test <<'END'
+1..2
+ok 1
+ok 2
END
-cat >>bar.test <<'END'
-#! /bin/sh
-echo "this is $0"
-sleep '1'
-exit 99
+
+cat > bar.test <<'END'
+1..1
+not ok 1
END
-cat >>baz.test <<'END'
-#! /bin/sh
-echo "this is $0"
-exit 1
+
+cat > baz.test <<'END'
+1..1
+Bail out!
END
-chmod a+x foo.test bar.test baz.test
$ACLOCAL
$AUTOCONF
@@ -62,97 +56,74 @@ $AUTOMAKE -a
./configure
-$MAKE check >stdout && { cat stdout; Exit 1; }
-cat stdout
-count_test_results total=3 pass=1 fail=1 skip=0 xfail=0 xpass=0 error=1
-test -f mylog.log
-cat mylog.log
-test `grep -c '^FAIL:' mylog.log` -eq 1
-test `grep -c '^ERROR:' mylog.log` -eq 1
-$EGREP '^(X?PASS|XFAIL|SKIP)' mylog.log && Exit 1
-test -f baz.log
-test -f bar.log
+# Even the tests that are not re-run should contribute to the testsuite
+# summary when obtained by "make check RECHECK_LOGS=".
+grep_summary ()
+{
+ grep '^# TOTAL: *4$' stdout
+ grep '^# PASS: *2$' stdout
+ grep '^# XPASS: *0$' stdout
+ grep '^# FAIL: *1$' stdout
+ grep '^# XFAIL: *0$' stdout
+ grep '^# SKIP: *0$' stdout
+ grep '^# ERROR: *1$' stdout
+}
+
+$MAKE -e check && Exit 1
test -f foo.log
+test -f bar.log
+test -f baz.log
-# clean should remove all log files (but not more).
-: > unrelated.log
-$MAKE clean
-test ! -f baz.log
-test ! -f bar.log
-test ! -f foo.log
-test ! -f mylog.log
-test -f unrelated.log
-
-# Check dependencies: baz.test needs to run before bar.test,
-# but foo.test is not needed.
-# FIXME: Note that this usage has a problem: the summary will only
-# take bar.log into account, because the $(TEST_SUITE_LOG) rule
-# does not "see" baz.log. Hmm.
-env TESTS='bar.test' $MAKE -e check >stdout && { cat stdout; Exit 1; }
-cat stdout
-grep '^FAIL: baz\.test$' stdout
-grep '^ERROR: bar\.test$' stdout
+rm -f foo.log bar.log
-test -f baz.log
-test -f bar.log
-test ! -f foo.log
-test -f mylog.log
-
-# Upon a lazy rerun, foo.test should be run, but the others shouldn't.
-# Note that the lazy rerun still exits with a failure, due to the previous
-# test failures.
-# Note that the previous test and this one taken together expose the timing
-# issue that requires the check-TESTS rule to always remove TEST_SUITE_LOG
-# before running the tests lazily.
env RECHECK_LOGS= $MAKE -e check > stdout && { cat stdout; Exit 1; }
cat stdout
test -f foo.log
-grep '^PASS: foo\.test$' stdout
-grep bar.test stdout && Exit 1
-grep baz.test stdout && Exit 1
-grep '^# PASS: *1$' stdout
-grep '^# FAIL: *1$' stdout
-grep '^# ERROR: *1$' stdout
-
-# Now, explicitly retry with all test logs already updated, and ensure
-# that the summary is still displayed.
+test -f bar.log
+grep '^PASS: foo\.test 1$' stdout
+grep '^PASS: foo\.test 2$' stdout
+grep '^FAIL: bar\.test' stdout
+grep 'baz\.test' stdout && Exit 1
+grep_summary
+
+$sleep
+touch foo.test
+# We re-run only a successful test, but the tests that failed in the
+# previous run should still be taken into account, and cause an overall
+# failure.
env RECHECK_LOGS= $MAKE -e check > stdout && { cat stdout; Exit 1; }
cat stdout
-grep foo.test stdout && Exit 1
-grep bar.test stdout && Exit 1
-grep baz.test stdout && Exit 1
-grep '^# PASS: *1$' stdout
-grep '^# FAIL: *1$' stdout
-grep '^# ERROR: *1$' stdout
-
-# Lazily rerunning only foo should only rerun this one test.
-env RECHECK_LOGS=foo.log $MAKE -e check > stdout && { cat stdout; Exit 1; }
-cat stdout
-grep foo.test stdout
-grep bar.test stdout && Exit 1
-grep baz.test stdout && Exit 1
-grep '^# PASS: *1$' stdout
-grep '^# FAIL: *1$' stdout
-grep '^# ERROR: *1$' stdout
-
-# Test VERBOSE.
-env VERBOSE=yes $MAKE -e check > stdout && { cat stdout; Exit 1; }
+grep '^PASS: foo\.test 1$' stdout
+grep '^PASS: foo\.test 2$' stdout
+grep 'ba[rz]\.test' stdout && Exit 1
+is_newest foo.log foo.test
+grep_summary
+
+$sleep
+touch zardoz
+env RECHECK_LOGS= $MAKE -e check > stdout && { cat stdout; Exit 1; }
cat stdout
-grep 'this is.*bar.test' stdout
-grep 'this is.*baz.test' stdout
+grep '^ERROR: baz\.test' stdout
+$EGREP '(foo|bar)\.test' stdout && Exit 1
+is_newest baz.log zardoz
+grep_summary
-$MAKE clean
-env TEST_LOGS=baz.log $MAKE -e check > stdout && { cat stdout; Exit 1; }
+# Now, explicitly retry with all test logs already updated, and ensure
+# that the summary is still displayed.
+env RECHECK_LOGS= $MAKE -e check > stdout && { cat stdout; Exit 1; }
cat stdout
-grep foo.test stdout && Exit 1
-grep bar.test stdout && Exit 1
-grep baz.test stdout
+$EGREP '(foo|bar|baz)\.test' stdout && Exit 1
+grep_summary
-$MAKE clean
-env TESTS=baz.test $MAKE -e check > stdout && { cat stdout; Exit 1; }
+# The following should re-run foo.test (and only foo.test), even if its
+# log file is up-to-date.
+: > older
+env RECHECK_LOGS=foo.log $MAKE -e check > stdout || { cat stdout; Exit 1; }
cat stdout
-grep foo.test stdout && Exit 1
-grep bar.test stdout && Exit 1
-grep baz.test stdout
+grep '^PASS: foo\.test 1$' stdout
+grep '^PASS: foo\.test 2$' stdout
+grep 'ba[rz]\.test' stdout && Exit 1
+is_newest foo.log older
+grep_summary
:
diff --git a/tests/tap-recheck.test b/tests/tap-recheck.test
index 21fec1e..68c830e 100755
--- a/tests/tap-recheck.test
+++ b/tests/tap-recheck.test
@@ -72,7 +72,7 @@ END
cat > d.test << 'END'
#! /bin/sh
-echo not ok 2 '# SKIP: who cares ...'
+echo ok 2 '# SKIP: who cares ...'
(. ./d.extra) || echo 'Bail out! d.extra failed'
: > d.run
END
diff --git a/tests/tap-skip-whole.test b/tests/tap-skip-whole.test
index d3d08b8..807c3f2 100755
--- a/tests/tap-skip-whole.test
+++ b/tests/tap-skip-whole.test
@@ -15,11 +15,13 @@
# along with this program. If not, see <http://www.gnu.org/licenses/>.
# Basic TAP test protocol support:
-# - special plan format to skip all the tests in a script.
+# - special plan format to skip all the tests in a script
parallel_tests=yes
. ./defs || Exit 1
+weirdchars=\''"$!&()[]<>#;^?*/@%=,.:'
+
cp "$top_testsrcdir"/lib/tap-driver . \
|| fatal_ "failed to fetch auxiliary script tap-driver"
@@ -31,7 +33,7 @@ END
cat > Makefile.am << 'END'
TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
TEST_LOG_COMPILER = cat
-TESTS = foo.test err.test wget.test curl.test
+TESTS = foo.test err.test wget.test curl.test mu.test
END
cat > foo.test <<END
@@ -53,11 +55,21 @@ ok 2 # SKIP who cares
END
cat > curl.test <<END
-1..0${tab}${tab} #${tab} ${tab}Can't connect to gnu.org!
+1..0${tab}${tab} #${tab} ${tab}Can't connect to gnu.org! ${tab}${tab}
not ok foo
ok bar # TODO but was supposed to fail
END
+cat > mu.test <<END
+1..0 #foo $weirdchars
+# TAP errors in here shouldn't be seen, as the test is skipped.
+1..2
+ok 1
+ok 22
+not ok 333
+1..7
+END
+
chmod a+x all.test
$ACLOCAL
@@ -73,7 +85,9 @@ count_test_results total=4 pass=0 fail=0 xpass=0 xfail=0
skip=4 error=0
grep '^SKIP: foo\.test$' stdout
grep '^SKIP: bar\.test$' stdout
-grep '^SKIP: wget\.test (wget not installed)' stdout
-grep '^SKIP: curl\.test (Can'\''t connect to gnu\.org!)$' stdout
+grep '^SKIP: wget\.test # wget(1) not installed$' stdout
+grep "^SKIP: curl\\.test #${tab} ${tab}Can't connect to gnu\\.org!
${tab}${tab}$" stdout
+grep "^SKIP: mu\\.test#foo $weirdchars$" stdout
+test `grep -c ': .*\.test' stdout` -eq 5
:
diff --git a/tests/tap-summary.test b/tests/tap-summary.test
index 5cc36f4..f902d5f 100755
--- a/tests/tap-summary.test
+++ b/tests/tap-summary.test
@@ -17,6 +17,7 @@
# TAP support:
# - testsuite summary
# - colorized testsuite summary
+# TODO: splitting this into two distinct test scripts might be worthwhile
parallel_tests=yes
. ./defs || Exit 1
@@ -24,11 +25,36 @@ parallel_tests=yes
cp "$top_testsrcdir"/lib/tap-driver . \
|| fatal_ "failed to fetch auxiliary script tap-driver"
-# The following shell variables are influential for this function_
+cat > configure.in <<END
+AC_INIT([GNU AutoTAP], [5.12], address@hidden)
+AM_INIT_AUTOMAKE([parallel-tests])
+AC_SUBST([PERL], ['$PERL'])
+AC_CONFIG_FILES([Makefile])
+AC_OUTPUT
+END
+
+cat > Makefile.stub << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TEST_LOG_COMPILER = cat
+TESTS = all.test
+END
+
+$ACLOCAL
+$AUTOCONF
+
+br='============================================================================'
+
+# The following shell variables are influential for this function:
# - expect_failure
# - use_colors
do_check ()
{
+ case $#,$1 in
+ 1,--pass) expect_failure=no;;
+ 1,--fail) expect_failure=yes;;
+ *) fatal_ "invalid usage of 'do_check'";;
+ esac
+ shift
cat > summary.exp
st=0
$MAKE check > stdout || st=$?
@@ -51,28 +77,284 @@ do_check ()
$compare summary.exp summary.got || Exit 1
}
-br='============================================================================'
+for use_colors in "yes" "no"; do
-cat > configure.in <<END
-AC_INIT([GNU AutoTAP], [5.12], address@hidden)
-AM_INIT_AUTOMAKE([parallel-tests])
-AC_SUBST([PERL], ['$PERL'])
-AC_CONFIG_FILES([Makefile])
-AC_OUTPUT
+ cp Makefile.stub Makefile.am
+
+ if test $use_colors = yes; then
+ red='[0;31m'
+ grn='[0;32m'
+ lgn='[1;32m'
+ blu='[1;34m'
+ mgn='[0;35m'
+ brg='[1m'
+ std='[m'
+ echo AUTOMAKE_OPTIONS = color-tests >> Makefile.am
+ else
+ red= grn= lgn= blu= mgn= brg= std=
+ fi
+
+ success_header="\
+${grn}${br}${std}
+${grn}Testsuite summary for GNU AutoFoo 7.1${std}
+${grn}${br}${std}"
+
+ success_footer=${grn}${br}${std}
+
+ failure_header="\
+${red}${br}${std}
+${red}Testsuite summary for GNU AutoFoo 7.1${std}
+${red}${br}${std}"
+
+ failure_footer="\
+${red}${br}${std}
+${red}See ./test-suite.log${std}
+${red}Please report to address@hidden
+${red}${br}${std}"
+
+ $AUTOMAKE --add-missing
+
+ ./configure
+
+ # 1 pass.
+ { echo 1..1 && echo ok; } > all.test
+ do_check --pass <<END
+$success_header
+${brg}# TOTAL: 1${std}
+${grn}# PASS: 1${std}
+# SKIP: 0
+# XFAIL: 0
+# FAIL: 0
+# XPASS: 0
+# ERROR: 0
+$success_footer
END
-cat > Makefile.am << 'END'
-TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
-TEST_LOG_COMPILER = cat
-TESTS = all.test
+ # 1 skip.
+ { echo 1..1 && echo 'ok # SKIP'; } > all.test
+ do_check --pass <<END
+$success_header
+${brg}# TOTAL: 1${std}
+# PASS: 0
+${blu}# SKIP: 1${std}
+# XFAIL: 0
+# FAIL: 0
+# XPASS: 0
+# ERROR: 0
+$success_footer
END
-$ACLOCAL
-$AUTOCONF
-$AUTOMAKE --add-missing
+ # 1 xfail.
+ { echo 1..1 && echo 'not ok # TODO'; } > all.test
+ do_check --pass <<END
+$success_header
+${brg}# TOTAL: 1${std}
+# PASS: 0
+# SKIP: 0
+${lgn}# XFAIL: 1${std}
+# FAIL: 0
+# XPASS: 0
+# ERROR: 0
+$success_footer
+END
+
+ # 1 fail.
+ { echo 1..1 && echo not ok; } > all.test
+ do_check --fail <<END
+$failure_header
+${brg}# TOTAL: 1${std}
+# PASS: 1
+# SKIP: 0
+# XFAIL: 0
+${red}# FAIL: 0${std}
+# XPASS: 0
+# ERROR: 0
+$failure_footer
+END
-./configure
+ # 1 xpass.
+ { echo 1..1 && echo 'ok # TODO'; } > all.test
+ do_check --fail <<END
+$failure_header
+${brg}# TOTAL: 1${std}
+# PASS: 0
+# SKIP: 0
+# XFAIL: 0
+# FAIL: 0
+${red}# XPASS: 1${std}
+# ERROR: 0
+$failure_footer
+END
+
+ # 1 hard error.
+ { echo 1..1 && echo 'Bail out!'; } > all.test
+ do_check --fail <<END
+$failure_header
+${brg}# TOTAL: 1${std}
+# PASS: 0
+# SKIP: 0
+# XFAIL: 0
+# FAIL: 0
+# XPASS: 0
+${mgn}# ERROR: 1${std}
+$failure_footer
+END
+
+ # 3 non-failing results.
+ cat > all.test <<END
+1..3
+ok
+not ok # TODO
+ok # SKIP
+END
+ do_check --pass <<END
+$success_header
+${brg}# TOTAL: 3${std}
+${grn}# PASS: 1${std}
+${blu}# SKIP: 1${std}
+${lgn}# XFAIL: 1${std}
+# FAIL: 0
+# XPASS: 0
+# ERROR: 0
+$success_footer
+END
+
+ # 1 pass, 1 skip, 1 fail.
+ cat > all.test <<END
+1..3
+ok
+ok # SKIP
+not ok
+END
+ do_check --fail <<END
+$failure_header
+${brg}# TOTAL: 3${std}
+${grn}# PASS: 1${std}
+${blu}# SKIP: 1${std}
+# XFAIL: 0
+${red}# FAIL: 1${std}
+# XPASS: 0
+# ERROR: 0
+$failure_footer
+END
+
+ # 1 pass, 1 xfail, 1 xpass.
+ cat > all.test <<END
+1..3
+ok
+ok # TODO
+not ok # TODO
+END
+ do_check --fail <<END
+$failure_header
+${brg}# TOTAL: 3${std}
+${grn}# PASS: 1${std}
+# SKIP: 0
+${lgn}# XFAIL: 1${std}
+# FAIL: 0
+${red}# XPASS: 1${std}
+# ERROR: 0
+$failure_footer
+END
+
+ # 1 skip, 1 xfail, 1 error.
+ cat > all.test <<END
+1..3
+ok # SKIP
+not ok # TODO
+Bail out!
+END
+ do_check --fail <<END
+$failure_header
+${brg}# TOTAL: 3${std}
+# PASS: 0
+${blu}# SKIP: 1${std}
+${lgn}# XFAIL: 1${std}
+# FAIL: 0
+# XPASS: 0
+${mgn}# ERROR: 1${std}
+$failure_footer
+END
+
+ # 1 of each kind
+ cat > all.test <<END
+1..6
+ok
+not ok
+ok # TODO
+not ok # TODO
+ok # SKIP
+Bail out!
+END
+ do_check --fail <<END
+$failure_header
+${brg}# TOTAL: 6${std}
+${grn}# PASS: 1${std}
+${blu}# SKIP: 1${std}
+${lgn}# XFAIL: 1${std}
+${red}# FAIL: 1${std}
+${red}# XPASS: 1${std}
+${mgn}# ERROR: 1${std}
+$failure_footer
+END
+
+ # Prepare some common data for later.
+ for i in 0 1 2 3 4 5 6 7 8 9; do
+ for j in 0 1 2 3 4 5 6 7 8 9; do
+ echo "ok"
+ echo "not ok # TODO"
+ echo "ok # SKIP"
+ done
+ done > tap
+
+ # Lots of non-failures (300 per kind).
+ (cat tap && cat tap && cat tap) > all.test
+ test `wc -l <all.test` -eq 900 || Exit 99 # Sanity check.
+ echo 1..900 >> all.test # Test plan.
+ do_check --fail <<END
+$success_header
+${brg}# TOTAL: 900${std}
+${grn}# PASS: 300${std}
+${blu}# SKIP: 300${std}
+${lgn}# XFAIL: 300${std}
+# FAIL: 0
+# XPASS: 0
+# ERROR: 0
+$success_footer
+END
+
+ # 1 failure and lots of non-failures means failure.
+ (cat tap && echo "not ok" && cat tap) > all.test
+ test `wc -l <all.test` -eq 601 || Exit 99 # Sanity check.
+ echo 1..601 >> all.test # Test plan.
+ do_check --fail <<END
+$failure_header
+${brg}# TOTAL: 601${std}
+${grn}# PASS: 200${std}
+${blu}# SKIP: 200${std}
+${lgn}# XFAIL: 200${std}
+${red}# FAIL: 1${std}
+# XPASS: 0
+# ERROR: 0
+$failure_footer
+END
+
+ # 1 error and lots of non-failures means failure.
+ (cat tap & sed 31q tap && echo 'Bail out!') > all.test
+ test `wc -l <all.test` -eq 331 || Exit 99 # Sanity check.
+ echo 1..331 >> all.test # Test plan.
+ do_check --fail <<END
+$failure_header
+${brg}# TOTAL: 331${std}
+${grn}# PASS: 110${std}
+${blu}# SKIP: 110${std}
+${lgn}# XFAIL: 110${std}
+# FAIL: 0
+# XPASS: 0
+${mgn}# ERROR: 1${std}
+$failure_footer
+END
-Exit 1; # TODO
+done # use_colors in "yes" "no"
:
diff --git a/tests/tap-todo-skip.test b/tests/tap-todo-skip.test
index 3bedf2a..bf95e37 100755
--- a/tests/tap-todo-skip.test
+++ b/tests/tap-todo-skip.test
@@ -112,7 +112,7 @@ count_test_results total=9 pass=9 fail=0 xpass=0 xfail=0
skip=0 error=0
# --------------------------------------------------------------------- #
cat > stub.tap <<END
-1..9
+1..15
ok 1 # TODO
ok 2 # TODO foo?
ok 3 # TODO: bar!
@@ -122,6 +122,12 @@ ok 6 cc # TODO:${tab}fnord 6
ok 7 - x # TODO
ok 8 - y # TODO fnord $tab 8
ok 9 - z # TODO: fnord 9 $tab
+ok 10# TODO 0
+ok 11$tab# TODO 1
+ok 12 $tab$tab # TODO 2
+ok 13 asd# TODO 3
+ok 14 sad$tab# TODO 4
+ok 15 das$tab$tab # TODO 5
END
cat > stub.exp <<END
@@ -134,6 +140,12 @@ cat > stub.exp <<END
: all.test 7 - x
: all.test 8 - y (fnord $tab 8)
: all.test 9 - z (fnord 9 $tab)
+: all.test 10 (0)
+: all.test 11 (1)
+: all.test 12 (2)
+: all.test 13 asd (3)
+: all.test 14 sad (4)
+: all.test 15 das (5)
END
# For "TODO" directives leading to XPASS results.
diff --git a/tests/tap-xfail-tests.test b/tests/tap-xfail-tests.test
index f8c4e09..4bf0913 100755
--- a/tests/tap-xfail-tests.test
+++ b/tests/tap-xfail-tests.test
@@ -14,8 +14,8 @@
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
-# TAP and $(XFAIL_TESTS): test results without directives are turned
-# from PASS to XPASS and from FAIL to XFAIL.
+# TAP and $(XFAIL_TESTS): test results without directives are turned from
+# PASS to XPASS and from FAIL to XFAIL; other results are unchanged.
parallel_tests=yes
. ./defs || Exit 1
@@ -35,6 +35,12 @@ TESTS = the.test
XFAIL_TESTS = $(TESTS)
END
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
cat > the.test <<END
1..6
ok 1
@@ -42,29 +48,40 @@ not ok 2
ok 3 # TODO
not ok 4 # TODO
ok 5 # SKIP
+# The next should count as a failure (and thus will be
+# rendered into an expected failure by XFAIL_TESTS).
not ok 6 # SKIP
Bail out!
END
-$ACLOCAL
-$AUTOCONF
-$AUTOMAKE
-
-./configure
-
$MAKE check >stdout && { cat stdout; Exit 1; }
cat stdout
-count_test_results total=7 pass=0 fail=0 xpass=2 xfail=2 skip=2 error=1
+count_test_results total=7 pass=0 fail=0 xpass=2 xfail=3 skip=1 error=1
grep '^XPASS: the\.test 1$' stdout
grep '^XFAIL: the\.test 2$' stdout
-grep '^XPASS: the\.test 3$' stdout
-grep '^XFAIL: the\.test 4$' stdout
-grep '^SKIP: the\.test 5$' stdout
-grep '^SKIP: the\.test 6$' stdout
+grep '^XPASS: the\.test 3 # TODO$' stdout
+grep '^XFAIL: the\.test 4 # TODO$' stdout
+grep '^SKIP: the\.test 5 # SKIP$' stdout
+grep '^XFAIL: the\.test 6 # SKIP$' stdout
grep '^ERROR: the\.test - Bail out!$' stdout
test `$FGREP -c ': the.test' stdout` -eq 7
+# Check that the exit status of the testsuite is indeed 0 when we
+# would expect success.
+
+cat > the.test <<END
+1..3
+not ok 1
+ok 2 # SKIP
+not ok 3 # TODO
+END
+
+$MAKE check >stdout || { cat stdout; Exit 1; }
+cat stdout
+
+count_test_results total=3 pass=0 fail=0 xpass=0 xfail=2 skip=1 error=0
+
: