[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[GSoC] preliminary tests on TAP support
|
From: |
Stefano Lattarini |
|
Subject: |
[GSoC] preliminary tests on TAP support |
|
Date: |
Thu, 7 Jul 2011 23:09:52 +0200 |
|
User-agent: |
KMail/1.13.3 (Linux/2.6.30-2-686; KDE/4.4.4; i686; ; ) |
Hello Automakers.
I've created a nice battery of tests on TAP support, and some sketchy
documentation. Some tests are still incomplete, but I think it's better
to start getting some early feedback anyway. Thus I've already pushed
them to the temporary branch 'GSoC/experimental/test-results-work' (note
that, since that branch is temporary and rewindable, the patch can still
be amended freely). Attached is what I've pushed.
As usual, reviews, suggestions and criticism are welcome. I should be
able to finish the incomplete tests by tomorrow evening.
Regards,
Stefano
From de73c975914eb119d8ba3370f0c40153f835c4a4 Mon Sep 17 00:00:00 2001
Message-Id: <address@hidden>
From: Stefano Lattarini <address@hidden>
Date: Sun, 3 Jul 2011 19:20:22 +0200
Subject: [PATCH] tap: add experimental TAP-aware driver
---
doc/automake.texi | 80 ++++++++-
lib/Makefile.am | 2 +-
lib/Makefile.in | 2 +-
lib/tap-driver | 80 +++++++++
tests/tap-autonumber.test | 93 ++++++++++
tests/tap-bailout.test | 118 +++++++++++++
tests/tap-basic.test | 74 ++++++++
tests/tap-color.test | 149 ++++++++++++++++
tests/tap-diagnostic.test | 129 ++++++++++++++
tests/tap-exit.test | 76 ++++++++
tests/tap-fancy.test | 94 ++++++++++
tests/tap-fancy2.test | 142 +++++++++++++++
tests/tap-global-result.test | 112 ++++++++++++
tests/tap-log.test | 124 +++++++++++++
tests/tap-merge-stdout-stderr.test | 72 ++++++++
tests/tap-more.test | 155 ++++++++++++++++
tests/tap-no-merge-stdout-stderr.test | 60 +++++++
tests/tap-no-ok-skip.test | 57 ++++++
tests/tap-no-spurious.test | 105 +++++++++++
tests/tap-numeric-description.test | 84 +++++++++
tests/tap-out-of-order.test | 93 ++++++++++
tests/tap-passthrough.test | 184 +++++++++++++++++++
tests/tap-plan-errors.test | 169 ++++++++++++++++++
tests/tap-plan.test | 89 ++++++++++
tests/tap-realtime.test | 99 +++++++++++
tests/tap-recheck-logs.test | 158 +++++++++++++++++
tests/tap-recheck.test | 214 +++++++++++++++++++++++
tests/tap-skip-whole.test | 79 +++++++++
tests/tap-summary.test | 78 ++++++++
tests/tap-todo-skip.test | 185 ++++++++++++++++++++
tests/tap-with-and-without-number.test | 51 ++++++
tests/tap-xfail-tests.test | 70 ++++++++
tests/test-driver-custom-multitest-recheck.test | 1 +
33 files changed, 3270 insertions(+), 8 deletions(-)
create mode 100755 lib/tap-driver
create mode 100755 tests/tap-autonumber.test
create mode 100755 tests/tap-bailout.test
create mode 100755 tests/tap-basic.test
create mode 100755 tests/tap-color.test
create mode 100755 tests/tap-diagnostic.test
create mode 100755 tests/tap-exit.test
create mode 100755 tests/tap-fancy.test
create mode 100755 tests/tap-fancy2.test
create mode 100755 tests/tap-global-result.test
create mode 100755 tests/tap-log.test
create mode 100755 tests/tap-merge-stdout-stderr.test
create mode 100755 tests/tap-more.test
create mode 100755 tests/tap-no-merge-stdout-stderr.test
create mode 100755 tests/tap-no-ok-skip.test
create mode 100755 tests/tap-no-spurious.test
create mode 100755 tests/tap-numeric-description.test
create mode 100755 tests/tap-out-of-order.test
create mode 100755 tests/tap-passthrough.test
create mode 100755 tests/tap-plan-errors.test
create mode 100755 tests/tap-plan.test
create mode 100755 tests/tap-realtime.test
create mode 100755 tests/tap-recheck-logs.test
create mode 100755 tests/tap-recheck.test
create mode 100755 tests/tap-skip-whole.test
create mode 100755 tests/tap-summary.test
create mode 100755 tests/tap-todo-skip.test
create mode 100755 tests/tap-with-and-without-number.test
create mode 100755 tests/tap-xfail-tests.test
diff --git a/doc/automake.texi b/doc/automake.texi
index 16804cd..9344beb 100644
--- a/doc/automake.texi
+++ b/doc/automake.texi
@@ -320,6 +320,7 @@ Support for test suites
* Simple Tests:: Listing programs and scripts in @code{TESTS}
* Simple Tests using parallel-tests:: More powerful test driver
* Custom Test Drivers:: Writing and using custom test drivers
+* Using the TAP test protocol:: Integrating test scripts that use the TAP
protocol
* DejaGnu Tests:: Interfacing with the external testing framework
* Install Tests:: Running tests on installed packages
@@ -8656,6 +8657,7 @@ are very similar.
* Simple Tests:: Listing programs and scripts in @code{TESTS}
* Simple Tests using parallel-tests:: More powerful test driver
* Custom Test Drivers:: Writing and using custom test drivers
+* Using the TAP test protocol:: Integrating test scripts that use the TAP
protocol
* DejaGnu Tests:: Interfacing with the external testing framework
* Install Tests:: Running tests on installed packages
@end menu
@@ -9037,12 +9039,9 @@ or work around the issue with inference rules to
generate the tests.
@subsection Overview of Custom Test Drivers Support
Starting form Automake version 1.12, the @option{parallel-tests} harness
-allows the package authors to use third-party custom test drivers, in case
-the
address@hidden FIXME: this should become "default ones" once we have TAP and
Subunit
-default one is inadequate for their purposes.
address@hidden FIXME: add this once we have TAP and Subunit
address@hidden "or do not support their testing protocol of choice."
+allows the package authors to use third-party custom test drivers, in
+case the default ones are inadequate for their purposes, or do not support
+their testing protocol of choice.
A custom test driver is expected to properly run the test programs
passed to it, including the command-line arguments passed to it,
@@ -9336,6 +9335,75 @@ is not the case, the HTML generation will not work,
although all the
other functionalities of the Automake testsuite harness should remain
untouched, and continue to work correctly.
address@hidden Using the TAP test protocol
address@hidden Using the TAP test protocol
+
+Brief introduction to TAP and its philosophy.
+
+Real-word uses of TAP: testsuite of @command{perl} and of many
+perl modules, and of @command{git}. Third-party libraries and
+utilities that can generate TAP (tell also to look at links
+below).
+
address@hidden
+A harness must only read TAP output from standard output and not
+from standard error. Lines written to standard output matching
address@hidden/^(not )?ok\b/} must be interpreted as test lines. All other
+lines must not be considered test output.
address@hidden quotation
+
+Links to more extensive official or third-party documentation and
+resources:
address@hidden @bullet
address@hidden
+The (mostly) official documentation about the TAP format and protocol:
address@hidden://search.cpan.org/~petdance/Test-Harness/lib/Test/Harness/TAP.pod}
address@hidden
+The most famous command-line TAP test driver, included in the distribution
+of @command{perl} and @samp{Test::Harness}:
address@hidden://search.cpan.org/~andya/Test-Harness/bin/prove}
address@hidden
+A wiki devoted to TAP:
address@hidden://testanything.org/wiki/index.php/Main_Page}
address@hidden
address@hidden: links to TAP-producing frameworks and libraries.
address@hidden itemize
+
+Give example of output from @command{prove} command (when used in
+``verbose'' mode, with option @option{-v}), to which we'd like to
+remain somewhat compatible compatible, while also (and foremost)
+being compatible with the default @option{parallel-tests} driver.
+
+Example of output from out own TAP testsuite driver. Point out
+similarities with the output from @command{prove --verbose} and
+from the default @option{parallel-tests} driver.
+
+Option @option{--ignore-exit} causes the driver to ignore the exit
+status of the test scripts; by default, the driver will report an
+error if the script exit with status != 0.
address@hidden: also add a @option{--no-ignore-exit} option, for
+completeness?
+
+By default, TAP diagnostic (i.e., lines beginning with the @samp{#}
+character) are copied only in the @file{.log} file. The option
address@hidden causes the driver to display them in the testsuite
+progress output too (@emph{TODO}: give example). The option
address@hidden restore the default behaviour.
+
+Differences and incompatibilities with other TAP parsers and runners:
address@hidden @bullet
address@hidden
+To avoid output synchronization problems, we read TAP directives not only
+from stdout, but also from stderr. This is similar to what happens using
+the @option{--merge} option of @command{prove}. @emph{FIXME}: maybe we
+should provide an option do disable this behaviour? I really think so.
address@hidden
+A @code{Bail out!} directive doesn't stop the whole testsuite, but only
+the test script it occurs into. This doesn't follows TAP specifications,
+but on the other hand maximize compatibility (and code sharing) with
+the ``hard error'' concept of the default @option{parallel-tests} driver.
address@hidden itemize
+
@node DejaGnu Tests
@section DejaGnu Tests
diff --git a/lib/Makefile.am b/lib/Makefile.am
index a26d663..a9467f1 100644
--- a/lib/Makefile.am
+++ b/lib/Makefile.am
@@ -29,7 +29,7 @@ dist_pkgvdata_DATA = COPYING INSTALL texinfo.tex ansi2knr.c
ansi2knr.1 \
scriptdir = $(pkgvdatadir)
dist_script_DATA = config.guess config.sub install-sh mdate-sh missing \
mkinstalldirs elisp-comp ylwrap acinstall depcomp compile py-compile \
- symlink-tree ar-lib test-driver
+ symlink-tree ar-lib test-driver tap-driver
EXTRA_DIST = gnupload
diff --git a/lib/Makefile.in b/lib/Makefile.in
index 5f8aa91..b600402 100644
--- a/lib/Makefile.in
+++ b/lib/Makefile.in
@@ -242,7 +242,7 @@ dist_pkgvdata_DATA = COPYING INSTALL texinfo.tex ansi2knr.c
ansi2knr.1 \
scriptdir = $(pkgvdatadir)
dist_script_DATA = config.guess config.sub install-sh mdate-sh missing \
mkinstalldirs elisp-comp ylwrap acinstall depcomp compile py-compile \
- symlink-tree ar-lib test-driver
+ symlink-tree ar-lib test-driver tap-driver
EXTRA_DIST = gnupload
all: all-recursive
diff --git a/lib/tap-driver b/lib/tap-driver
new file mode 100755
index 0000000..e94d82f
--- /dev/null
+++ b/lib/tap-driver
@@ -0,0 +1,80 @@
+#! /usr/bin/env perl
+# Temporary/experimental TAP test driver for Automake.
+# TO be rewritten portably (e.g., in awk or shell).
+
+use warnings FATAL => 'all';
+use strict;
+use Getopt::Long ();
+use TAP::Parser;
+
+my $me = "tap-driver";
+
+my $usage = <<'END';
+Usage:
+ tap-driver [--help|--version] --test-name=NAME --log-file=PATH
+ [--expect-failure={yes|no}] [--color-tests={yes|no}]
+ [--enable-hard-errors={yes|no}] [--] TEST-COMMAND
+The `--test-name' and `--log-file' options are mandatory.
+END
+
+my $help = <<END;
+$me: TAP-aware test driver for Automake testsuite harness.
+$usage;
+END
+
+my %cfg = (
+ "color-tests" => 0,
+ "expect-failure" => 0,
+ "enable-hard-errors" => 1,
+);
+
+my $test_name = undef;
+my $log_file = undef;
+
+sub bool_opt ($$)
+{
+ my ($opt, $val) = @_;
+ if ($val =~ /^(?:y|yes)\z/i)
+ {
+ $cfg{opt} = 1;
+ }
+ elsif ($val =~ /^(?:n|no)\z/i)
+ {
+ $cfg{opt} = 1;
+ }
+ else
+ {
+ die "invalid argument '$val' for option '$opt'\n";
+ }
+}
+
+my %cli_options =
+ (
+ 'help' => sub { print $help; },
+ 'test-name=s' => \$test_name,
+ 'log-file=s' => \$log_file,
+ 'color-tests=s' => \&bool_opt,
+ 'expect-failure=s' => \&bool_opt,
+ 'enable-hard-errors=s' => \&bool_opt,
+ );
+
+Getopt::Long::GetOptions %cli_options or exit 1;
+
+#my %tap_callbacks = (
+# test => \&test_callback,
+# plan => \&plan_callback,
+# comment => \&comment_callback,
+# bailout => \&bailout_callback,
+# unknown => \&unknown_callback,
+#);
+
+open(LOG, ">$log_file") or die "opening $log_file: $!";
+
+print "@ARGV\n";
+my $parser = TAP::Parser->new( { exec => address@hidden } );
+
+while (my $result = $parser->next) {
+ print LOG $result->as_string . "\n";
+}
+
+# vim: ft=perl ts=4 sw=4 et
diff --git a/tests/tap-autonumber.test b/tests/tap-autonumber.test
new file mode 100755
index 0000000..6b77484
--- /dev/null
+++ b/tests/tap-autonumber.test
@@ -0,0 +1,93 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support:
+# - unnumbered tests are OK, as long as their final total number
+# agrees with the plan
+# - test results without number get automatically numbered in the
+# console progress output
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TESTS = all.test
+EXTRA_DIST = $(TESTS)
+END
+
+cat > all.test <<'END'
+1..14
+ok
+ok foo
+ok - foo2
+not ok
+not ok bar
+not ok - bar2
+; See that we can intermingle different kind of results without
+; messing up the autonumbering
+ok
+ok # TODO
+not ok # TODO who cares?
+ok
+not ok
+ok muuu # SKIP
+not ok
+ok
+END
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+$MAKE check >stdout && { cat stdout; Exit 1; }
+cat stdout
+count_test_results total=14 pass=6 fail=5 xpass=1 xfail=1 skip=1 error=0
+
+cat > exp <<'END'
+PASS: all.test 1
+PASS: all.test 2 foo
+PASS: all.test 3 - foo2
+FAIL: all.test 4
+FAIL: all.test 5 bar
+FAIL: all.test 6 - bar2
+PASS: all.test 7
+XPASS: all.test 8
+XFAIL: all.test 9 (who cares?)
+PASS: all.test 10
+FAIL: all.test 11
+SKIP: all.test 12 muu
+FAIL: all.test 13
+PASS: all.test 14
+END
+
+$FGREP ': all.test' stdout > got
+
+cat exp
+cat got
+diff exp got
+
+:
diff --git a/tests/tap-bailout.test b/tests/tap-bailout.test
new file mode 100755
index 0000000..442c274
--- /dev/null
+++ b/tests/tap-bailout.test
@@ -0,0 +1,118 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# Basic TAP test protocol support:
+# - "Bail out!" magic
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TEST_LOG_COMPILER = cat
+TESTS = a.test b.test c.test d.test e.test
+END
+
+# pass += 1, fail +=1, error += 1
+cat > a.test <<END
+1..4
+ok 1
+not ok 2
+Bail out!
+not ok 3
+ok 4 # SKIP
+END
+
+# skip += 1, xpass += 1, xfail += 1, error += 1
+cat > b.test <<END
+1..7
+ok 1 # SKIP
+ok 2 # TODO
+not ok 3 # TODO
+Bail out!${tab}${sp}${sp}${tab}We're out of disk space.
+ok 4
+not ok 5
+not ok 6 # TODO
+ok 7 # TODO
+END
+
+# Bail out before the test plan.
+# pass += 2, error += 1
+cat > c.test <<END
+ok 1
+ok 2
+Bail out! ... BOOOH!${tab}${sp}
+1..2
+END
+
+# TAP input comprised only of a bailout directive.
+# error += 1
+cat > d.test <<END
+Bail out!${tab}${sp}
+END
+
+# Bailout on the first line.
+# error += 1
+cat > e.test <<END
+Bail out! mktemp -d: Permission denied
+ok 1
+END
+
+# Doing the sums above, we have:
+test_counts='total=11 pass=3 fail=1 xpass=1 xfail=1 skip=1 error=5'
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+$MAKE check >stdout || { cat stdout; Exit 1; }
+cat stdout
+
+count_test_results $test_counts
+
+grep '^PASS: a\.test 1$' stdout
+grep '^FAIL: a\.test 2$' stdout
+grep '^ERROR: a\.test - Bail out!$' stdout
+test `$FGREP -c ': a.test' stdout` -eq 3
+
+grep '^SKIP: b\.test 1$' stdout
+grep '^XPASS: b\.test 2$' stdout
+grep '^XFAIL: b\.test 3$' stdout
+grep '^ERROR: b\.test - Bail out! We'\''re out of disk space\.$' stdout
+test `$FGREP -c ': b.test' stdout` -eq 4
+
+grep '^PASS: c\.test 1$' stdout
+grep '^PASS: c\.test 2$' stdout
+grep '^ERROR: c\.test - Bail out! \.\.\. BOOOH!'"${tab}${sp}$" stdout
+test `$FGREP -c ': c.test' stdout` -eq 3
+
+grep '^ERROR: d\.test - Bail out!$' stdout
+test `$FGREP -c ': d.test' stdout` -eq 1
+
+grep '^ERROR: e\.test - Bail out! mktemp -d: Permission denied$' stdout
+test `$FGREP -c ': e.test' stdout` -eq 1
+
+:
diff --git a/tests/tap-basic.test b/tests/tap-basic.test
new file mode 100755
index 0000000..2ee8786
--- /dev/null
+++ b/tests/tap-basic.test
@@ -0,0 +1,74 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# Basic TAP support:
+# - LOG_COMPILER support
+# - basic support for TODO and SKIP directives
+# - testsuite progress output on console
+# - runtime override of TESTS and TEST_LOGS
+# - correct counts of test results (both in summary and in
+# progress output on console)
+# Note that some of the features checked here are checked in other
+# test cases too, usually in more detail.
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+## Defining LOG_COMPILER should work and not intefere with the
+## tap-driver script.
+TEST_LOG_COMPILER = cat
+TESTS = all.test
+END
+
+metacharacters=\''"\$!&()[]<>#;^?*'
+
+cat > all.test <<END
+1..23
+ok 1
+ok 2 foo
+ok 3 - foofoo
+ok 4 bar baz
+not ok 5 xfoo
+not ok 6 - xfoo
+not ok 7 xbar xbaz
+not ok 8 xfoo
+ok 5 $metacharacters
+END
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+$MAKE check >stdout && { cat stdout; Exit 1; }
+cat stdout
+
+count_test_results total=10 pass=2 fail=2 xpass=2 xfail=2 skip=2 error=0
+
+Exit 1; # TODO ALL
+
+:
diff --git a/tests/tap-color.test b/tests/tap-color.test
new file mode 100755
index 0000000..f4a5f21
--- /dev/null
+++ b/tests/tap-color.test
@@ -0,0 +1,149 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support:
+# - colorization of TAP results and diagnostic messages
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+TERM=ansi; export TERM
+
+esc=''
+# Escape `[' for grep, below.
+red="$esc\[0;31m"
+grn="$esc\[0;32m"
+lgn="$esc\[1;32m"
+blu="$esc\[1;34m"
+mgn="$esc\[0;35m"
+std="$esc\[m"
+
+
+# Check that grep can parse nonprinting characters.
+# BSD 'grep' works from a pipe, but not a seekable file.
+# GNU or BSD 'grep -a' works on files, but is not portable.
+case `echo "$std" | grep .` in
+ "$std") ;;
+ *) echo "$me: grep can't parse nonprinting characters" >&2; Exit 77;;
+esac
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+AUTOMAKE_OPTIONS = color-tests
+AM_TEST_LOG_DRIVER_FLAGS = --comments
+TEST_LOG_COMPILER = cat
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TESTS = all.test skip.test bailout.test plan-errors.test
+END
+
+cat > all.test << 'END'
+1..5
+ok 1 - foo
+# Hi! I shouldn't be colorized!
+not ok 2 - bar # TODO difficult
+ok 3 - baz # SKIP I hate baz
+not ok 4 - quux
+ok 5 - zardoz # TODO
+END
+
+cat > skip.test << 'END'
+1..0 # skip whole script
+END
+
+cat > errors.test << 'END'
+1..1
+ok 1
+Bail out!
+END
+
+# Remember to test also test that get completely skipped:
+# plan: 1..0
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+AM_COLOR_TESTS=always $MAKE check >stdout && { cat stdout; Exit 1; }
+cat stdout
+
+test_color ()
+{
+ # Not a useless use of cat; see above comments about grep.
+ cat stdout | grep "^${grn}PASS${std}: all\.test 1 - foo$"
+ cat stdout | grep "^${lgn}XFAIL${std}: all\.test 2 - bar (difficult)$"
+ cat stdout | grep "^${blu}SKIP${std}: all\.test 3 - baz (I hate baz)$"
+ cat stdout | grep "^${red}FAIL${std}: all\.test 4 - quux$"
+ cat stdout | grep "^${red}XPASS${std}: all\.test 5 - zardoz$"
+ cat stdout | grep "^${blu}SKIP${std}: skip\.test (skip whole script)$"
+ cat stdout | grep "^${grn}PASS${std}: bailout\.test 1$"
+ cat stdout | grep "^${mgn}ERROR${std}: all\.test - Bail out!$"
+ # Diagnostic messages shouldn't be colored.
+ cat stdout | grep "^DIAG: all\.test # Hi! I shouldn't be colorized!$"
+ :
+}
+
+test_no_color ()
+{
+ # With make implementations that, like Solaris make, in case of errors
+ # print the whole failing recipe on standard output, we should content
+ # ourselves with a laxer check, to avoid false positives.
+ # Keep this in sync with lib/am/check.am:$(am__color_tests).
+ if $FGREP '= Xalways || test -t 1 ' stdout; then
+ # Extra verbose make, resort to laxer checks.
+ # But we also want to check that the testsuite summary is not unduly
+ # colorized.
+ (
+ set +e # In case some grepped regex below isn't matched.
+ # Not a useless use of cat; see above comments about grep.
+ cat stdout | grep "TOTAL.*:"
+ cat stdout | grep "PASS.*:"
+ cat stdout | grep "FAIL.*:"
+ cat stdout | grep "SKIP.*:"
+ cat stdout | grep "XFAIL.*:"
+ cat stdout | grep "XPASS.*:"
+ cat stdout | grep "ERROR.*:"
+ cat stdout | grep "DIAG.*:"
+ cat stdout | grep 'test.*expected'
+ cat stdout | grep 'test.*not run'
+ cat stdout | grep '===='
+ cat stdout | grep '[Ss]ee .*test-suite\.log'
+ cat stdout | grep '[Tt]estsuite summary'
+ ) | grep "$esc" && Exit 1
+ : For shells with broken 'set -e'
+ else
+ cat stdout | grep "$esc" && Exit 1
+ : For shells with broken 'set -e'
+ fi
+}
+
+AM_COLOR_TESTS=always $MAKE check >stdout && { cat stdout; Exit 1; }
+cat stdout
+test_color
+
+$MAKE -e check >stdout && { cat stdout; Exit 1; }
+cat stdout
+test_no_color
+
+:
diff --git a/tests/tap-diagnostic.test b/tests/tap-diagnostic.test
new file mode 100755
index 0000000..9988f1e
--- /dev/null
+++ b/tests/tap-diagnostic.test
@@ -0,0 +1,129 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support:
+# - diagnostic messages (TAP lines with leading "#")
+# - flags '--comments' and '--no-comments' of the TAP test driver
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TEST_LOG_COMPILER = cat
+TESTS = all.test
+END
+
+sp=' '
+tab=' '
+metacharacters=\''"\$!&()[]<>#;^?*'
+
+cat > all.test <<END
+1..3
+# Hi! I'm a comment.
+# Tests begin.
+ok 1
+not ok 2 - foo # TODO
+ok 3 - bar # SKIP
+# Tests end.
+#No leading whitespace is OK.
+# ${tab}${tab} ${tab}More leading whitespace is kept.
+# Trailing whitespace should be kept, 1.$sp
+# Trailing whitespace should be kept, 2.$tab
+# All${tab}${tab}whitespace should ${tab} be kept. $tab
+# Shell metacharacters here: $metacharacters
+ok - zardoz
+# Following lines shouldn't be parsed as test results.
+# ok
+#ok
+# not ok
+#not ok
+# Bail out!
+#Bail out!
+# SKIP
+#SKIP
+# TODO
+#TODO
+.# Leading characters before "#", not a TAP diagnostic line.
+x # Leading characters before "#", not a TAP diagnostic line.
+ # Leading whitespace before "#", not a TAP diagnostic line.
+${tab}# Leading whitespace before "#", not a TAP diagnostic line.
+ ${tab} # Leading whitespace before "#", not a TAP diagnostic line.
+END
+
+cat > exp <<END
+DIAG: all.test # Hi! I'm a comment.
+DIAG: all.test # Tests begin.
+PASS: all.test 1
+XFAIL: all.test 2 - foo
+SKIP: all.test 3 - bar
+DIAG: all.test # Tests end.
+DIAG: all.test #No leading whitespace is OK.
+DIAG: all.test # ${tab}${tab} ${tab}More leading whitespace is kept.
+DIAG: all.test # Leading whitespace should be stripped, 3.
+DIAG: all.test # Trailing whitespace should be kept, 1.$sp
+DIAG: all.test # Trailing whitespace should be kept, 2.$tab
+DIAG: all.test # All${tab}${tab}whitespace should ${tab} be kept. $tab
+DIAG: all.test # Shell metacharacters here: $metacharacters
+PASS: all.test - 4 - zardoz
+DIAG: all.test # Following lines shouldn't be parsed as test results.
+DIAG: all.test # ok
+DIAG: all.test #ok
+DIAG: all.test # not ok
+DIAG: all.test #not ok
+DIAG: all.test # Bail out!
+DIAG: all.test #Bail out!
+DIAG: all.test # SKIP
+DIAG: all.test #SKIP
+DIAG: all.test # TODO
+DIAG: all.test #TODO
+END
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+$MAKE check >stdout || { cat stdout; Exit 1; }
+cat stdout
+grep '^DIAG:' stdout && Exit 1
+count_test_results total=3 pass=3 fail=0 xpass=0 xfail=1 skip=1 error=0
+
+echo 'AM_TEST_LOG_DRIVER_FLAGS = --comments' >> Makefile
+$MAKE check >stdout || { cat stdout; Exit 1; }
+cat stdout
+LC_ALL=C grep '^[A-Z][A-Z]*:' stdout > got
+cat exp
+cat got
+diff exp got
+count_test_results total=3 pass=3 fail=0 xpass=0 xfail=1 skip=1 error=0
+
+TEST_LOG_DRIVER_FLAGS="--no-comments" $MAKE -e check >stdout \
+ || { cat stdout; Exit 1; }
+cat stdout
+grep '^DIAG:' stdout && Exit 1
+count_test_results total=3 pass=3 fail=0 xpass=0 xfail=1 skip=1 error=0
+
+:
diff --git a/tests/tap-exit.test b/tests/tap-exit.test
new file mode 100755
index 0000000..be464a5
--- /dev/null
+++ b/tests/tap-exit.test
@@ -0,0 +1,76 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support:
+# - an exit status != 0 of a test script causes an hard error
+# - the `--ignore-exit' option causes the TAP test driver to ignore
+# exit statuses of the test scripts.
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TESTS = signal.test
+END
+
+cat > signal.test <<'END'
+#! /bin/sh
+echo 1..1
+echo ok 1
+kill $$
+END
+
+for st in 1 2 77 99; do
+ unindent > exit$i.test <<END
+ #!/bin/sh
+ echo 1..1
+ echo ok 1
+ exit $st
+END
+ echo TESTS += exit$i.test >> Makefile.am
+done
+
+chmod a+x *.test
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+$MAKE check >stdout && { cat stdout; Exit 1; }
+cat stdout
+count_test_results total=10 pass=5 fail=0 xpass=0 xfail=0 skip=0 error=5
+
+grep '^ERROR: exit1\.test - exited with status 1$' stdout
+grep '^ERROR: exit2\.test - exited with status 2$' stdout
+grep '^ERROR: exit77\.test - exited with status 77$' stdout
+grep '^ERROR: exit99\.test - exited with status 99$' stdout
+grep '^ERROR: signal\.test - terminated by signal.*' stdout
+
+echo TEST_LOG_DRIVER_FLAGS = --ignore-exit >> Makefile
+$MAKE check >stdout && { cat stdout; Exit 1; }
+cat stdout
+count_test_results total=5 pass=5 fail=0 xpass=0 xfail=0 skip=0 error=0
+
+:
diff --git a/tests/tap-fancy.test b/tests/tap-fancy.test
new file mode 100755
index 0000000..96403a5
--- /dev/null
+++ b/tests/tap-fancy.test
@@ -0,0 +1,94 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support: some unusual forms for valid TAP input.
+# See also related test 'tap-fancy2.test'.
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TEST_LOG_COMPILER = cat
+TESTS = all.test
+END
+
+#
+# From manpage Test::Harness::TAP(3):
+#
+# ``Lines written to standard output matching /^(not )?ok\b/ must be
+# interpreted as test lines. All other lines must not be considered
+# test output.''
+#
+# Unfortunately, the exact format of TODO and SKIP directives is not as
+# clearly described in that manpage; but a simple reverse-engineering of
+# the prove(1) utility shows that it is probably given by the perl regex
+# /#\s*(TODO|SKIP)\b/.
+#
+
+cat > all.test <<END
+1..19
+
+ok? a question
+not ok? a question
+
+ok+plus
+not ok+plus
+
+ok-minus
+not ok-minus
+
+ok#55
+not ok#55
+
+ok${tab} ${tab}9
+ok ${tab}${tab} 10
+
+not ok${tab} ${tab}11
+not ok ${tab}${tab} 12
+
+ok# SKIP
+ok${tab}#SKIP--who cares?
+ok?#SKIP!
+
+not ok# TODO
+not ok${tab}#TODO--who cares?
+ok?#TODO!
+
+ok~~#TODO~~
+END
+
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+$MAKE check >stdout && { cat stdout; Exit 1; }
+cat stdout
+
+count_test_results total=19 pass=6 fail=6 xfail=3 xpass=1 skip=3
+
+:
diff --git a/tests/tap-fancy2.test b/tests/tap-fancy2.test
new file mode 100755
index 0000000..ae5e0ef
--- /dev/null
+++ b/tests/tap-fancy2.test
@@ -0,0 +1,142 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support: more unusual forms for valid TAP input.
+# See also related test 'tap-fancy.test'.
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TEST_LOG_COMPILER = cat
+TESTS = all.test
+END
+
+#
+# From manpage Test::Harness::TAP(3):
+#
+# ``Lines written to standard output matching /^(not )?ok\b/ must be
+# interpreted as test lines. All other lines must not be considered
+# test output.''
+#
+# Unfortunately, the exact format of TODO and SKIP directives is not as
+# clearly described in that manpage; but a simple reverse-engineering of
+# the prove(1) utility shows that it is probably given by the perl regex
+# /#\s*(TODO|SKIP)\b/.
+#
+
+# There are 32 values for $str ...
+for str in \
+ \' \
+ '"' \
+ '`' \
+ '#' \
+ '$' \
+ '!' \
+ '\' \
+ '&' \
+ '%' \
+ '(' \
+ ')' \
+ '|' \
+ '^' \
+ '~' \
+ '?' \
+ '*' \
+ '+' \
+ '-' \
+ ',' \
+ ':' \
+ ';' \
+ '=' \
+ '<' \
+ '>' \
+ '@' \
+ '[' \
+ ']' \
+ '{' \
+ '}' \
+ '...' \
+ '?[a-zA-Z0-9]*' \
+ '*.*' \
+; do
+ # ... each of them add 1 pass, 1 fail, ...
+ echo "ok${str}"
+ echo "not ok${str}"
+ # ... and (generally) 4 skips, 4 xfails, and 4 xpasses ...
+ for settings in \
+ 'result="ok" directive=SKIP' \
+ 'result="not ok" directive=TODO' \
+ 'result="ok" directive=TODO' \
+ ; do
+ eval "$settings"
+ echo "${result}# ${directive}${str}"
+ # ... but 3 skips, 3 xpasses and 3 xfails are to be removed, since
+ # they don't work with $str = "#" ...
+ if test x"$str" != x"#"; then
+ echo "${result}${str}#${directive}"
+ echo "${result}${str}# ${tab}${tab} ${directive}"
+ echo "${result}${str}#${directive}${str}"
+ fi
+ done
+done > all.test
+
+# ... so that we finally have:
+pass=32
+fail=32
+xfail=125 # = 4 * 32 - 3
+xpass=125 # = 4 * 32 - 3
+skip=125 # = 4 * 32 - 3
+error=0
+total=`expr $pass + $fail + $xfail + $xpass + $skip`
+
+# Even nastier!
+for result in 'ok' 'not ok'; do
+ echo "${result}{[(<#${tab}TODO>)]}" >> all.test
+done
+echo "ok{[(<#${tab}SKIP>)]}" >> all.test
+
+# We have to update the some test counts.
+xfail=`expr $xfail + 1`
+xpass=`expr $xpass + 1`
+skip=`expr $skips + 1`
+total=`expr $total + 3`
+
+# And add the test plan!
+echo 1..$total >> all.test
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+$MAKE check >stdout && { cat stdout; Exit 1; }
+cat stdout
+
+count_test_results total=$total pass=$pass fail=$fail skip=$skip \
+ xpass=$xpass xfail=$xfail error=$error
+
+:
diff --git a/tests/tap-global-result.test b/tests/tap-global-result.test
new file mode 100755
index 0000000..ca36b9c
--- /dev/null
+++ b/tests/tap-global-result.test
@@ -0,0 +1,112 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support:
+# - which global test result derives from different test results
+# mixed in a single script?
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TEST_LOG_COMPILER = cat
+TESTS = \
+ ok.test \
+ fail.test fail2.test \
+ xpass.test xpass2.test \
+ error.test error2.test \
+ hodgepodge.test hodgepodge-all.test
+END
+
+cat > ok.test <<END
+1..3
+ok 1
+not ok 2 # TODO
+ok 3 # SKIP
+END
+
+cat > fail.test <<'END'
+1..1
+not ok 1
+END
+
+(sed '1s/.*/1..4/' ok.test && echo 'not ok 4') > fail2.test
+
+cat > xpass.test <<'END'
+1..1
+ok 1 # TODO
+END
+
+(sed '1s/.*/1..4/' ok.test && echo 'ok 4 # TODO') > xpass2.test
+
+echo 'Bail out!' > error.test
+(cat ok.test && echo 'Bail out!') > error2.test
+
+cat > hodgepodge.test <<'END'
+1..2
+not ok 1
+ok 2 # TODO
+Bail out!
+END
+
+cat > hodgepodge-all.test <<'END'
+1..4
+ok 1
+ok 2 # SKIP
+not ok 2 # TODO
+not ok 3
+ok 4 # TODO
+Bail out!
+END
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+$MAKE check stdout && { cat stdout; Exit 1; }
+cat stdout
+
+count_test_results total=27 pass=5 fail=4 xpass=4 xfail=5 skip=5 error=4
+
+for tst in ok fail fail2 xpass xpass2 error error2 \
+ hodgepodge hodgepodge-all; do
+ sed -e 2q $tst > $tst.res
+done
+
+cat *.res # For debugging.
+
+grep '^PASS:' ok.res
+grep '^FAIL:' fail.res
+grep '^FAIL:' fail2.res
+grep '^FAIL:' xpass.res
+grep '^FAIL:' xpass2.res
+grep '^ERROR:' error.res
+grep '^ERROR:' error2.res
+grep '^ERROR:' hodgepodge.res
+grep '^ERROR:' hodgepodge-all.res
+
+:
diff --git a/tests/tap-log.test b/tests/tap-log.test
new file mode 100755
index 0000000..7bccd3c
--- /dev/null
+++ b/tests/tap-log.test
@@ -0,0 +1,124 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support:
+# - log file creation
+# - log file removal
+# - TEST_SUITE_LOG redefinition, at either automake or make time
+# - VERBOSE environment variable support
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TEST_LOG_COMPILER = cat
+TESTS = pass.test skip.test xfail.test fail.test xpass.test error.test
+TEST_SUITE_LOG = global.log
+END
+
+marker=%%% # Custom marker, for use in grepping checks.
+
+cat > pass.test <<END
+1..1
+# $marker pass $marker
+ok 1
+END
+
+cat > skip.test <<END
+1..1
+# $marker skip $marker
+not ok 1 # SKIP
+END
+
+cat > xfail.test <<END
+1..1
+# $marker xfail $marker
+not ok 1 # TODO
+END
+
+cat > fail.test <<END
+1..1
+# $marker fail $marker
+not ok 1
+END
+
+cat > pass.test <<END
+1..1
+# $marker xpass $marker
+ok 1 # TODO
+END
+
+cat > error.test <<END
+1..1
+Bail out!
+END
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+TEST_SUITE_LOG=my.log $MAKE -e check && Exit 1
+ls -l # For debugging.
+test ! -f test-suite.log
+test ! -f global.log
+test -f my.log
+for result in pass fail xfail xpass skip error; do
+ $FGREP "$marker $result $marker" $result.log
+done
+$FGREP "$marker skip $marker" my.log
+$FGREP "$marker fail $marker" my.log
+$FGREP "$marker error $marker" my.log
+test `$FGREP -c "$marker" my.log` -eq 3
+
+touch error2.log test-suite.log global.log
+$MAKE mostlyclean
+ls -l # For debugging.
+test ! -f my.log
+test ! -f pass.log
+test ! -f fail.log
+test ! -f xfail.log
+test ! -f xpass.log
+test ! -f skip.log
+test ! -f error.log
+test -f error2.log
+test -f test-suite.log
+test -f global.log
+
+rm -f *.log
+
+VERBOSE=yes $MAKE check >stdout && { cat stdout; Exit 1; }
+cat stdout
+cat global.log
+test ! -f my.log
+test ! -f test-suite.log
+# Check that VERBOSE causes the global testsuite log to be
+# emitted on stdout.
+out=`cat stdout`
+log=`cat global.log`
+case $out in *"$log"*) ;; *) Exit 1;; esac
+
+:
diff --git a/tests/tap-merge-stdout-stderr.test
b/tests/tap-merge-stdout-stderr.test
new file mode 100755
index 0000000..ef50818
--- /dev/null
+++ b/tests/tap-merge-stdout-stderr.test
@@ -0,0 +1,72 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support:
+# - The Automake TAP driver has an option that instruct it to read TAP
+# input only from the stdout of the test command, not also its stderr
+# (as is required by the TAP specifications).
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TESTS = all.test
+END
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+cat > all.test <<END
+#!/bin/sh
+echo 1..4
+echo ok 1 >&2
+echo ok 2
+echo "not ok 3 # TODO" >&2
+echo "not ok 4 # SKIP"
+echo "# foo foo foo" >&2
+END
+
+$MAKE check >stdout || { cat stdout; Exit 1; }
+cat stdout
+
+count_test_results total=4 pass=2 fail=0 xpass=0 xfail=1 skip=1 error=0
+grep '^DIAG:.*foo foo foo' stdout
+
+cat > all.test <<END
+#!/bin/sh
+echo 1..1
+echo ok 1
+echo 'Bail out!' >&2
+END
+
+$MAKE check >stdout && { cat stdout; Exit 1; }
+cat stdout
+
+count_test_results total=2 pass=1 fail=0 xpass=0 xfail=0 skip=0 error=1
+
+:
diff --git a/tests/tap-more.test b/tests/tap-more.test
new file mode 100755
index 0000000..345d7e1
--- /dev/null
+++ b/tests/tap-more.test
@@ -0,0 +1,155 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# More on TAP support:
+# - more than one TAP-generating test script in $(TESTS)
+# - VPATH builds
+# - use with parallel make (if supported)
+# - basic use of diagnostic messages (lines beginning with "#")
+# - messages generated by the testsuite harness reference the
+# correct test script(s)
+# - "make distcheck" works
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TESTS = 1.test 2.test 3.test
+EXTRA_DIST = $(TESTS)
+END
+
+cat > 1.test <<'END'
+#! /bin/sh
+echo 1..2
+echo ok 1 mu
+if test -f not-skip; then
+ echo "not ok 2 zardoz"
+else
+ echo "not ok 2 zardoz # SKIP"
+fi
+END
+
+cat > 2.test <<'END'
+#! /bin/sh
+echo 1..3
+echo "ok"
+echo "not ok # TODO not implemented"
+echo "ok 3
+END
+
+cat > 3.test <<END
+#! /bin/sh
+echo 1..1
+echo ok - blah blah blah
+echo '# Some diagnostic'
+if echo test -f bail-out; then
+ echo 'Bail out! Kernel Panic'
+else
+ :
+fi
+END
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+# Try a VPATH and by default serial build first, and then an in-tree
+# and by default parallel build.
+for try in 0 1; do
+
+ if test $try -eq 0; then
+ # VPATH serial build.
+ mkdir build
+ cd build
+ srcdir=..
+ run_make=$MAKE
+ elif test $try -eq 1; then
+ # In-tree parallel build.
+ srcdir=.
+ case $MAKE in
+ *\ -j*)
+ # Degree of parallelism already specified by the user: do
+ # not override it.
+ run_make=$MAKE
+ ;;
+ *)
+ # Some make implementations (e.g., HP-UX) don't grok `-j',
+ # some require no space between `-j' and the number of jobs
+ # (e.g., older GNU make versions), and some *do* require a
+ # space between `-j' and the number of jobs (e.g., Solaris
+ # dmake). We need a runtime test to see what works.
+ echo 'all:' > Makefile
+ for run_make in "$MAKE -j3" "$MAKE -j 3" "$MAKE"; do
+ $run_make && break
+ done
+ rm -f Makefile
+ ;;
+ esac
+ else
+ fatal_ "internal error, invalid value of '$try' for \$try"
+ fi
+
+ $srcdir/configure
+
+ # Success.
+ $run_make check >stdout || { cat stdout; Exit 1; }
+ cat stdout
+ count_test_results total=6 pass=4 fail=0 xpass=0 xfail=1 skip=1 error=0
+ grep '^PASS: 1\.test 1 - mu$' stdout
+ grep '^SKIP: 1\.test 2 - zardoz$' stdout
+ test `$FGREP -c : 1.test stdout` -eq 2
+ grep '^PASS: 2\.test 1$' stdout
+ grep '^XFAIL: 2\.test 2 (not implemented)$' stdout
+ grep '^PASS: 2\.test 3$' stdout
+ test `$FGREP -c : 2.test stdout` -eq 3
+ grep '^PASS: 3\.test 1 - blah blah blah$' stdout
+ grep '^DIAG: 3\.test # Some diagnostic$' stdout
+ test `$FGREP -c : 3.test stdout` -eq 2
+
+ # Failure.
+ : > not-skip
+ : > bail-out
+ $run_make check >stdout && { cat stdout; Exit 1; }
+ cat stdout
+ count_test_results total=7 pass=3 fail=1 xpass=0 xfail=1 skip=0 error=1
+ grep '^PASS: 1\.test 1 - mu$' stdout
+ grep '^FAIL: 1\.test 2 - zardoz$' stdout
+ test `$FGREP -c : 1.test stdout` -eq 2
+ grep '^PASS: 2\.test 1$' stdout
+ grep '^XFAIL: 2\.test 2 (not implemented)$' stdout
+ grep '^PASS: 2\.test 3$' stdout
+ test `$FGREP -c : 2.test stdout` -eq 3
+ grep '^PASS: 3\.test 1 - blah blah blah$' stdout
+ grep '^DIAG: 3\.test # Some diagnostic$' stdout
+ grep '^ERROR: 3\.test - Bail out! Kernel Panic$' stdout
+ test `$FGREP -c : 3.test stdout` -eq 3
+
+ cd $srcdir
+
+done
+
+$MAKE distcheck
+
+:
diff --git a/tests/tap-no-merge-stdout-stderr.test
b/tests/tap-no-merge-stdout-stderr.test
new file mode 100755
index 0000000..8083c93
--- /dev/null
+++ b/tests/tap-no-merge-stdout-stderr.test
@@ -0,0 +1,60 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support:
+# - By default, TAP input is read from both the stderr and stdout of the
+# test command (similarly with what happens with "prove --merge").
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TESTS = all.test
+END
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+cat > all.test <<END
+#!/bin/sh
+echo 1..2
+echo 'not ok' >&2
+echo 'ok 1'
+echo '# foo foo foo' >&2
+echo 'ok 2 # TODO' >&2
+echo 'Bail out!' >&2
+echo 'ok 2 # SKIP'
+END
+
+$MAKE check >stdout || { cat stdout; Exit 1; }
+cat stdout
+
+count_test_results total=2 pass=1 fail=0 xpass=0 xfail=0 skip=1 error=0
+$EGREP '(DIAG:|foo foo foo)' stdout && Exit 1
+
+:
diff --git a/tests/tap-no-ok-skip.test b/tests/tap-no-ok-skip.test
new file mode 100755
index 0000000..57b8d7a
--- /dev/null
+++ b/tests/tap-no-ok-skip.test
@@ -0,0 +1,57 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support:
+# - a "not ok # SKIP" line should count as a failure, for consistency
+# with the prove(1) utility.
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TEST_LOG_COMPILER = cat
+TESTS = all.test
+END
+
+cat > all.test <<'END'
+1..4
+not ok # SKIP
+not ok 2 # SKIP
+not ok - foo # SKIP
+not ok 4 - bar # SKIP
+END
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+$MAKE check >stdout && { cat stdout; Exit 1; }
+cat stdout
+
+count_test_results total=4 pass=0 fail=4 skip=0 xpass=0 xfail=0 error=0
+
+:
diff --git a/tests/tap-no-spurious.test b/tests/tap-no-spurious.test
new file mode 100755
index 0000000..ba1586b
--- /dev/null
+++ b/tests/tap-no-spurious.test
@@ -0,0 +1,105 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support:
+# - don't spuriously recognize lines that are "almost" TAP lines as
+# real TAP lines
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TEST_LOG_COMPILER = cat
+TESTS = all.test
+END
+
+echo 1..3 > all.test
+
+# The only recognized directives are "TODO" and "SKIP".
+# So the following should count as passed tests.
+cat >> all.test <<END
+ok 1 # XFAIL
+ok 2 # SKIPPED
+ok 3 # TODOALL
+END
+
+# According to documentation of Test::Harness::TAP(3)
+# ``Lines written to standard output matching /^(not )?ok\b/
+# must be interpreted as test lines. All other lines must
+# not be considered test output.''
+
+cat >> all.test <<END
+ ok
+ ok 4
+${tab}ok
+${tab}ok 6
+ok7
+ not ok
+ not ok 9
+${tab}not ok
+${tab}not ok 11
+not ok12
+notok
+notok 14
+not${tab}ok
+not${tab}ok 16
+not ok
+not ok 18
+no ok
+no ok 20
+# ok
+# not ok
+# ok 23
+# not ok 24
+END
+
+# The prove(1) utility doesn't bail out on these, so our driver
+# shouldn't either.
+cat >> all.test <<'END'
+bailout
+bailout!
+bail out
+bail out!
+Bailout
+Bailout!
+Bail out
+ Bail out
+ Bail out!
+#Bail out!
+# Bail out!
+END
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+$MAKE check >stdout && { cat stdout; Exit 1; }
+cat stdout
+
+count_test_results total=3 pass=3 fail=0 xpass=0 xfail=0 skip=0 error=0
+
+:
diff --git a/tests/tap-numeric-description.test
b/tests/tap-numeric-description.test
new file mode 100755
index 0000000..5275b77
--- /dev/null
+++ b/tests/tap-numeric-description.test
@@ -0,0 +1,84 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP result lines whose description is a number.
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TEST_LOG_COMPILER = cat
+TESTS = all.test
+END
+
+# Some random numbers to be used as test names.
+# These are selected so that $n != n for every n >= 1.
+set 5 7 45 11 7 3 6 9 1000 1
+# $1 $2 $3 $4 $5 $6 $7 $8 $9 $10
+
+cat > all.test <<END
+1..10
+ok - ${1}
+ok 2 ${2}
+not ok 3 ${3}
+not ok - ${4}
+ok 5 ${5} # SKIP
+ok - ${6} # SKIP
+not ok 7 ${7} # TODO
+not ok - ${8} # TODO
+ok 9 ${9} # TODO
+ok - 10 ${10} # TODO
+END
+
+chmod a+x all.test
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+$MAKE check >stdout && { cat stdout; Exit 1; }
+cat stdout
+
+count_test_results total=10 pass=2 fail=2 xpass=2 xfail=2 skip=2 error=0
+
+# Usage: do grep RESULT NUM DESC
+do_grep () { grep "^$1: all\\.test $2 - $3$" stdout; }
+
+do_grep PASS 1 $1
+do_grep PASS 2 $2
+do_grep FAIL 3 $3
+do_grep FAIL 4 $4
+do_grep SKIP 5 $5
+do_grep SKIP 6 $6
+do_grep XPASS 7 $7
+do_grep XPASS 8 $8
+do_grep XFAIL 9 $9
+do_grep XFAIL 10 ${10}
+
+$EGREP ": all\.test([^:]|.*\[|$)" stdout && Exit 1
+
+:
diff --git a/tests/tap-out-of-order.test b/tests/tap-out-of-order.test
new file mode 100755
index 0000000..df71983
--- /dev/null
+++ b/tests/tap-out-of-order.test
@@ -0,0 +1,93 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support:
+# - out-of-order test results
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TEST_LOG_COMPILER = cat
+TESTS = a.test b.test c.test d.test
+END
+
+cat > a.test <<END
+1..3
+ok
+not ok 2 # TODO
+ok 71
+END
+
+cat > b.test <<END
+1..5
+ok 1
+ok 2
+ok 4 foo # SKIP
+not ok 2 # TODO
+ok 5
+END
+
+cat > c.test <<END
+1..4
+ok 1
+ok foo
+ok 4 - bar
+not ok 3
+END
+
+# This is documented as an "test out of sequence" error in the
+# TAP::Parser documentation. Keep it in sync with the example
+# there.
+cat > d.test <<'END'
+ok 1
+ok 2
+ok 15
+ok 16
+ok 17
+END
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+$MAKE check >stdout && { cat stdout; Exit 1; }
+cat stdout
+
+count_test_results total=18 pass=11 fail=0 xpass=0 xfail=1 skip=0 error=6
+test `$FGREP -c ': a.test' stdout` -eq 3
+test `$FGREP -c ': b.test' stdout` -eq 5
+test `$FGREP -c ': c.test' stdout` -eq 4
+test `$FGREP -c ': d.test' stdout` -eq 5
+
+grep '^ERROR: a\.test - test out of order (expecting 3, got 71)$'
+grep '^ERROR: b\.test - test out of order (expecting 3, got 4)$'
+grep '^ERROR: b\.test - test out of order (expecting 4, got 2)$'
+grep '^ERROR: c\.test - test out of order (expecting 3, got 4)$'
+grep '^ERROR: d\.test - test out of order (expecting 3, got 15)$'
+
+:
diff --git a/tests/tap-passthrough.test b/tests/tap-passthrough.test
new file mode 100755
index 0000000..21ad5ad
--- /dev/null
+++ b/tests/tap-passthrough.test
@@ -0,0 +1,184 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support:
+# - all input (valid TAP lines, invalid TAP lines, non-TAP lines)
+# are passed through in the log file
+# - TAP parse errors are reported in the log file too.
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TEST_LOG_COMPILER = cat
+TESTS =
+END
+
+weirdchars=\''"address@hidden&()[]<>#;,:.^?*/'
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+#
+# Only successful tests.
+#
+
+cat > ok.test <<END
+1..6
+TAP plan in the previous line.
+ok${tab}
+ok 2
+ok - foo
+ok 4 - x
+ This is not a TAP line, but should still be copied in the log file!
+not ok # TODO low priority
+not ok # SKIP who cares?
+This is not in test-suite.log, since ok.test does not fail.
+$weirdchars
+END
+
+st=0
+TESTSL=ok.test $MAKE -e check || st=$?
+cat ok.log
+cat test-suite.log
+test $st -eq 0 || Exit 1
+
+for rx in \
+ '1\.\.6' \
+ 'TAP plan in the previous line\.' \
+ "ok${tab}" \
+ 'ok 2' \
+ 'ok - foo' \
+ 'ok 4 - x' \
+ 'This is not a TAP line, but should still be copied in the log file!' \
+ 'not ok # TODO low priority' \
+ 'not ok # SKIP who cares?' \
+ 'This is not in test-suite\.log, since ok\.test does not fail\.' \
+; do
+ grep "^$rx$" foo.log
+done
+$FGREP "$weirdchars" foo.log
+
+$FGREP 'foo.test' test-suite.log && Exit 1
+$FGREP -i 'this is not in test-suite.log' test-suite.log && Exit 1
+
+#
+# Mixed failing/successful tests.
+#
+
+cat > ok.test <<END
+1..1
+ok
+local only
+END
+
+cat > ko.test <<END
+1..5
+foo foo foo
+ok${tab}
+ok 2
+not ok - foo
+not ok 4 - x
+ bar${tab}bar${tab}bar
+ok # TODO dunno
+$weirdchars
+END
+
+cat > bail.test <<END
+Bail out! Test is taking too long!
+END
+
+cat > skip.test <<END
+1..0 # Skipped: WWW::Mechanize not installed
+END
+
+cat > err.test <<END
+1..3
+ok 1
+Invalid test count
+ok 23
+Misplaced plan
+1..3
+ok
+Extra test
+ok
+Last line
+END
+
+st=0
+TESTS='ok.test ko.test bail.test skip.test err.test' $MAKE -e check || st=$?
+cat ok.log
+cat ko.log
+cat bail.log
+cat skip.log
+cat err.log
+cat test-suite.log
+test $st -gt 0 || Exit 1
+
+grep '^local only$' ok.log
+grep 'local only' test-suite.log && Exit 1
+
+for rx in \
+ '1\.\.5' \
+ 'foo foo foo' \
+ "ok${tab}" \
+ 'ok 2' \
+ 'not ok - foo' \
+ 'not ok 4 - x' \
+ " bar${tab}bar${tab}bar" \
+ 'ok # TODO dunno' \
+; do
+ grep "^$rx$" ko.log
+ grep "^$rx$" test-suite.log
+done
+$FGREP "$weirdchars" ko.log
+$FGREP "$weirdchars" test-suite.log
+
+for log in bail.log test-suite; do
+ grep '^Bail out! Test is taking too long!$' $log
+done
+for log in skip.log test-suite; do
+ grep '^1\.\.0 # Skipped: WWW::Mechanize not installed$' $log
+done
+
+for rx in \
+ '^Invalid test count$' \
+ '^Misplaced plan$' \
+ '^Extra test$' \
+ '^Last line$' \
+ '^ok 23$' \
+ '[tT]oo many tests run.*expected 3, got 4' \
+ '[oO]ut of order.*expecting 2, got 23'
+; do
+ grep "$rx" err.log
+ grep "$rx" test-suite.log
+done
+test `grep -c '^1\.\.3$' err.log` -eq 2
+test `grep -c '^1\.\.3$' test-suite.log` -eq 2
+
+:
diff --git a/tests/tap-plan-errors.test b/tests/tap-plan-errors.test
new file mode 100755
index 0000000..3e92f48
--- /dev/null
+++ b/tests/tap-plan-errors.test
@@ -0,0 +1,169 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support: the following are flagged as errors:
+# - unmatched test plan
+# - misplaced test plan
+# - multiple test plans
+# - missing test plan
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TEST_LOG_COMPILER = cat
+TESTS = foo.test
+END
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+my_check ()
+{
+ cat > foo.test
+ $MAKE check >stdout && { cat stdout; Exit 1; }
+ cat stdout
+ count_test_results "$@"
+}
+
+cat > foo.test <<END
+1..2
+ok 1
+END
+my_check pass=1 fail=0 xpass=0 xfail=0 skip=0 error=1
+grep '^ERROR: foo\.test - too few tests run (expected 2, got 1)$' stdout
+
+my_check pass=2 fail=0 xpass=0 xfail=0 skip=0 error=1 <<END
+ok 1
+ok 2
+not ok 3 # TODO
+1..12
+END
+grep '^ERROR: foo\.test - too few tests run (expected 12, got 3)$' stdout
+
+my_check pass=0 fail=0 xpass=0 xfail=0 skip=0 error=1 <<END
+1..1
+END
+grep '^ERROR: foo\.test - too few tests run (expected 1, got 0)$' stdout
+
+my_check pass=2 fail=0 xpass=0 xfail=0 skip=0 error=1 <<END
+1..1
+ok 1
+ok 2
+END
+grep '^ERROR: foo\.test - too many tests run (expected 1, got 2)$' stdout
+
+my_check pass=4 fail=0 xpass=0 xfail=0 skip=5 error=1 <<END
+1..4
+ok 1
+ok 2
+ok 3
+ok 4
+ok 5
+ok 6
+ok 7
+END
+grep '^ERROR: foo\.test - too many tests run (expected 4, got 7)$' stdout
+
+my_check pass=2 fail=0 xpass=0 xfail=1 skip=0 error=1 <<END
+ok 1
+ok 2
+not ok 3 # TODO
+1..2
+END
+grep '^ERROR: foo\.test - too many tests run (expected 2, got 3)$' stdout
+
+# Leading blank line is meant.
+my_check pass=2 fail=0 xpass=0 xfail=0 skip=0 error=1 <<END
+
+1..2
+ok 1
+ok 2
+END
+grep '^ERROR: foo\.test - test plan in middle of output$' stdout
+
+# Trailing blank line is meant.
+my_check pass=2 fail=0 xpass=0 xfail=0 skip=0 error=1 <<END
+ok 1
+ok 2
+1..2
+
+END
+grep '^ERROR: foo\.test - test plan in middle of output$' stdout
+
+# In the next three checks, the two plans are equal -- this is deliberate.
+
+my_check pass=2 fail=0 xpass=0 xfail=0 skip=0 error=1 <<END
+1..1
+1..1
+ok 1
+END
+grep '^ERROR: foo\.test - multiple test plans$' stdout
+
+my_check pass=2 fail=0 xpass=0 xfail=0 skip=0 error=1 <<END
+ok 1
+1..1
+1..1
+END
+grep '^ERROR: foo\.test - multiple test plans$' stdout
+
+my_check pass=2 fail=0 xpass=0 xfail=0 skip=0 error=1 <<END
+1..2
+ok 1
+ok 2
+1..2
+END
+grep '^ERROR: foo\.test - multiple test plans$' stdout
+
+my_check pass=1 fail=0 xpass=0 xfail=0 skip=0 error=1 <<END
+ok 1
+END
+grep '^ERROR: foo\.test - missing test plans$' stdout
+
+my_check pass=0 fail=0 xpass=0 xfail=0 skip=0 error=1 </dev/null
+grep '^ERROR: foo\.test - missing test plans$' stdout
+
+# Check error message in corner cases (1).
+my_check pass=2 fail=0 xpass=0 xfail=0 skip=0 error=1 <<END
+1..2
+ok 1
+1..2
+ok 2
+END
+grep '^ERROR: foo\.test - test plan in middle of output$' stdout
+
+# Check error message in corner cases (2).
+my_check pass=2 fail=0 xpass=0 xfail=0 skip=0 error=1 <<END
+ok 1
+1..2
+ok 2
+1..2
+END
+grep '^ERROR: foo\.test - test plan in middle of output$' stdout
+
+:
diff --git a/tests/tap-plan.test b/tests/tap-plan.test
new file mode 100755
index 0000000..6183204
--- /dev/null
+++ b/tests/tap-plan.test
@@ -0,0 +1,89 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support:
+# - test scripts with the test plan at the beginning
+# - test scripts with the test plan at the end
+# - test scripts without a test plan
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TEST_LOG_COMPILER = cat
+TESTS = top.test bot.test none.test
+END
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+cat > top.test <<END
+1..3
+ok 1
+not ok 2 # SKIP
+ok 3
+# a tailing comment
+END
+
+cat > bot.test <<END
+ok 1
+# a comment
+ok 2
+not ok 3 # TODO
+# another comment
+ok
+1..4
+END
+
+cat > none.test <<END
+# 1..3
+# the previous line shouldn't be picked up by mistake.
+ok 1
+ok 2
+END
+
+# Check that the plans doesn't cause any problem
+
+$MAKE check >stdout || { cat stdout; Exit 1; }
+cat stdout
+count_test_results total=9 pass=7 xfail=1 skip=1 fail=0 xpass=0 error=0
+
+# Now check that the plans had been really considered.
+
+sed '1s/.*/1..9/' top.test > t
+mv -f t top.test
+sed '$s/.*/1..9/' bot.test > t
+mv -f t bot.test
+
+$MAKE check >stdout || { cat stdout; Exit 1; }
+cat stdout
+count_test_results total=11 pass=7 xfail=1 skip=1 fail=0 xpass=0 error=2
+grep '^ERROR: top\.test' stdout
+grep '^ERROR: bot\.test' stdout
+
+:
diff --git a/tests/tap-realtime.test b/tests/tap-realtime.test
new file mode 100755
index 0000000..ab26084
--- /dev/null
+++ b/tests/tap-realtime.test
@@ -0,0 +1,99 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support:
+# - testsuite progress on console should happen mostly "in real time";
+# i.e., it's not acceptable for the driver to wait the end of the
+# script to start displaying results from it.
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+# Check that we have a working expect program.
+cat >expect-check <<'END'
+eval spawn $env(SHELL) -c "echo ok"
+expect "^ok$"
+END
+expect -f expect-check || {
+ echo "$me: failed to find a working expect program" >&2
+ Exit 77
+}
+rm -f expect-check
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TESTS = all.test
+END
+
+cat > all.test <<'END'
+#! /bin/sh
+echo 1..9
+echo ok 1
+echo ok 2
+echo ok 3
+: > milestone-1
+sleep '20' # FIXME: quoting required to placate maintainer-check
+echo ok 4
+echo ok 5
+echo ok 6
+: > milestone-2
+sleep '20' # FIXME: quoting required to placate maintainer-check
+echo ok 7
+echo ok 8
+echo ok 9
+END
+
+chmod a+x all.test
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+cat >expect-make <<'END'
+eval spawn $env(MAKE) -e check
+expect eof
+END
+
+# Expect should simulate a tty as stdout, which should ensure a
+# line-buffered output.
+MAKE=$MAKE expect -f expect-make > stdout &
+
+while test ! -f milestone-1; do sleep '1'; done
+cat stdout
+test `$FGREP ': all.test' stdout` -eq 3
+for i in 1 2 3; do grep "^PASS: all\.test $i$"; done
+
+while test ! -f milestone-2; do sleep '1'; done
+cat stdout
+test `$FGREP ': all.test' stdout` -eq 6
+for i in 1 2 3 4 5 6; do grep "^PASS: all\.test $i$"; done
+
+wait
+cat stdout
+test `$FGREP ': all.test' stdout` -eq 9
+for i in 1 2 3 4 5 6 7 8 9; do grep "^PASS: all\.test $i$"; done
+
+:
diff --git a/tests/tap-recheck-logs.test b/tests/tap-recheck-logs.test
new file mode 100755
index 0000000..01ff155
--- /dev/null
+++ b/tests/tap-recheck-logs.test
@@ -0,0 +1,158 @@
+#! /bin/sh
+# Copyright (C) 2009, 2010, 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support:
+# - RECHECK_LOGS
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+Exit 1 # TODO
+
+cat >> configure.in << 'END'
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_SUITE_LOG = mylog.log
+TESTS = foo.test bar.test baz.test
+XFAIL_TESTS = bar.test
+foo.log: bar.log
+bar.log: baz.log
+END
+
+# foo.test and bar.test sleep to ensure their logs are always strictly newer
+# than the logs of their prerequisites, for HP-UX make. The quoting pleases
+# maintainer-check.
+cat >>foo.test <<'END'
+#! /bin/sh
+echo "this is $0"
+sleep '1'
+exit 0
+END
+cat >>bar.test <<'END'
+#! /bin/sh
+echo "this is $0"
+sleep '1'
+exit 99
+END
+cat >>baz.test <<'END'
+#! /bin/sh
+echo "this is $0"
+exit 1
+END
+chmod a+x foo.test bar.test baz.test
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE -a
+
+./configure
+
+$MAKE check >stdout && { cat stdout; Exit 1; }
+cat stdout
+count_test_results total=3 pass=1 fail=1 skip=0 xfail=0 xpass=0 error=1
+test -f mylog.log
+cat mylog.log
+test `grep -c '^FAIL:' mylog.log` -eq 1
+test `grep -c '^ERROR:' mylog.log` -eq 1
+$EGREP '^(X?PASS|XFAIL|SKIP)' mylog.log && Exit 1
+test -f baz.log
+test -f bar.log
+test -f foo.log
+
+# clean should remove all log files (but not more).
+: > unrelated.log
+$MAKE clean
+test ! -f baz.log
+test ! -f bar.log
+test ! -f foo.log
+test ! -f mylog.log
+test -f unrelated.log
+
+# Check dependencies: baz.test needs to run before bar.test,
+# but foo.test is not needed.
+# FIXME: Note that this usage has a problem: the summary will only
+# take bar.log into account, because the $(TEST_SUITE_LOG) rule
+# does not "see" baz.log. Hmm.
+env TESTS='bar.test' $MAKE -e check >stdout && { cat stdout; Exit 1; }
+cat stdout
+grep '^FAIL: baz\.test$' stdout
+grep '^ERROR: bar\.test$' stdout
+
+test -f baz.log
+test -f bar.log
+test ! -f foo.log
+test -f mylog.log
+
+# Upon a lazy rerun, foo.test should be run, but the others shouldn't.
+# Note that the lazy rerun still exits with a failure, due to the previous
+# test failures.
+# Note that the previous test and this one taken together expose the timing
+# issue that requires the check-TESTS rule to always remove TEST_SUITE_LOG
+# before running the tests lazily.
+env RECHECK_LOGS= $MAKE -e check > stdout && { cat stdout; Exit 1; }
+cat stdout
+test -f foo.log
+grep '^PASS: foo\.test$' stdout
+grep bar.test stdout && Exit 1
+grep baz.test stdout && Exit 1
+grep '^# PASS: *1$' stdout
+grep '^# FAIL: *1$' stdout
+grep '^# ERROR: *1$' stdout
+
+# Now, explicitly retry with all test logs already updated, and ensure
+# that the summary is still displayed.
+env RECHECK_LOGS= $MAKE -e check > stdout && { cat stdout; Exit 1; }
+cat stdout
+grep foo.test stdout && Exit 1
+grep bar.test stdout && Exit 1
+grep baz.test stdout && Exit 1
+grep '^# PASS: *1$' stdout
+grep '^# FAIL: *1$' stdout
+grep '^# ERROR: *1$' stdout
+
+# Lazily rerunning only foo should only rerun this one test.
+env RECHECK_LOGS=foo.log $MAKE -e check > stdout && { cat stdout; Exit 1; }
+cat stdout
+grep foo.test stdout
+grep bar.test stdout && Exit 1
+grep baz.test stdout && Exit 1
+grep '^# PASS: *1$' stdout
+grep '^# FAIL: *1$' stdout
+grep '^# ERROR: *1$' stdout
+
+# Test VERBOSE.
+env VERBOSE=yes $MAKE -e check > stdout && { cat stdout; Exit 1; }
+cat stdout
+grep 'this is.*bar.test' stdout
+grep 'this is.*baz.test' stdout
+
+$MAKE clean
+env TEST_LOGS=baz.log $MAKE -e check > stdout && { cat stdout; Exit 1; }
+cat stdout
+grep foo.test stdout && Exit 1
+grep bar.test stdout && Exit 1
+grep baz.test stdout
+
+$MAKE clean
+env TESTS=baz.test $MAKE -e check > stdout && { cat stdout; Exit 1; }
+cat stdout
+grep foo.test stdout && Exit 1
+grep bar.test stdout && Exit 1
+grep baz.test stdout
+
+:
diff --git a/tests/tap-recheck.test b/tests/tap-recheck.test
new file mode 100755
index 0000000..21fec1e
--- /dev/null
+++ b/tests/tap-recheck.test
@@ -0,0 +1,214 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# Test the 'recheck' target for TAP test protocol.
+# Keep in sync with 'test-driver-custom-multitest-recheck.test'.
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TESTS = a.test b.test c.test d.test
+END
+
+cat > a.test << 'END'
+#! /bin/sh
+echo 1..2
+echo ok 1
+echo ok 2
+: > a.run
+END
+
+cat > b.test << 'END'
+#! /bin/sh
+echo 1..2
+echo ok
+if test -f b.ok; then
+ echo ok - 2
+else
+ echo 'Bail out!'
+fi
+: > b.run
+END
+
+cat > c.test << 'END'
+#! /bin/sh
+echo 1..2
+if test -f c.pass; then
+ echo 'ok - c is ok :-)'
+else
+ echo 'not ok - c is ko :-('
+fi
+if test -f c.xfail; then
+ echo 'not ok 2 # TODO'
+else
+ echo 'ok 2 # TODO'
+fi
+echo 'not ok 3 - blah blah # TODO need better diagnostic'
+: > c.run
+END
+
+cat > d.test << 'END'
+#! /bin/sh
+echo not ok 2 '# SKIP: who cares ...'
+(. ./d.extra) || echo 'Bail out! d.extra failed'
+: > d.run
+END
+
+chmod a+x *.test
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+do_recheck ()
+{
+ case $* in
+ --fail) on_bad_rc='&&';;
+ --pass) on_bad_rc='||';;
+ *) fatal_ "invalid usage of function 'do_recheck'";;
+ esac
+ rm -f *.run
+ eval "\$MAKE recheck >stdout $on_bad_rc { cat stdout; ls -l; Exit 1; }; :"
+ cat stdout; ls -l
+}
+
+for vpath in : false; do
+ if $vpath; then
+ mkdir build
+ cd build
+ srcdir=..
+ else
+ srcdir=.
+ fi
+
+ $srcdir/configure
+
+ : A "make recheck" in a clean tree should run no tests.
+ do_recheck --pass
+ cat test-suite.log
+ test ! -r a.run
+ test ! -r a.log
+ test ! -r b.run
+ test ! -r b.log
+ test ! -r c.run
+ test ! -r c.log
+ test ! -r d.run
+ test ! -r d.log
+ count_test_results total=0 pass=0 fail=0 xpass=0 xfail=0 skip=0 error=0
+
+ : Run the tests for the first time.
+ $MAKE check >stdout && { cat stdout; Exit 1; }
+ cat stdout
+ ls -l
+ # All the test scripts should have run.
+ test -f a.run
+ test -f b.run
+ test -f c.run
+ test -f d.run
+ count_test_results total=9 pass=3 fail=2 xpass=1 xfail=1 skip=1 error=1
+
+ : Let us make b.test pass.
+ echo OK > b.ok
+ do_recheck --fail
+ # a.test has been successful the first time, so no need to re-run it.
+ # Similar considerations apply to similar checks, below.
+ test ! -r a.run
+ test -f b.run
+ test -f c.run
+ test -f d.run
+ count_test_results total=7 pass=2 fail=2 xpass=1 xfail=1 skip=1 error=0
+
+ : Let us make the first part of c.test pass.
+ echo OK > c.pass
+ do_recheck --fail
+ test ! -r a.run
+ test ! -r b.run
+ test -f c.run
+ test -f d.run
+ count_test_results total=5 pass=1 fail=1 xpass=1 xfail=1 skip=1 error=0
+
+ : Let us make also the second part of c.test pass.
+ echo KO > c.xfail
+ do_recheck --fail
+ test ! -r a.run
+ test ! -r b.run
+ test -f c.run
+ test -f d.run
+ count_test_results total=5 pass=1 fail=1 xpass=0 xfail=2 skip=1 error=0
+
+ : Nothing changed, so only d.test should be run.
+ for i in 1 2; do
+ do_recheck --fail
+ test ! -r a.run
+ test ! -r b.run
+ test ! -r c.run
+ test -f d.run
+ count_test_results total=2 pass=0 fail=1 xpass=0 xfail=0 skip=1 error=0
+ done
+
+ : Let us make d.test run more testcases, and experience _more_ failures.
+ unindent > d.extra <<'END'
+ echo 'ok # SKIP s'
+ echo 'not ok - f 1'
+ echo 'ok - p 1'
+ echo 'not ok - f 2'
+ echo 'ok # TODO xp'
+ echo 'not ok - f 3'
+ echo 'not ok - f 4'
+ # Misplaced plan
+ echo '1..4'
+ echo 'ok - p 2'
+ echo 'Bail out!'
+END
+ do_recheck --fail
+ test ! -r a.run
+ test ! -r b.run
+ test ! -r c.run
+ test -f d.run
+ count_test_results total=11 pass=2 fail=4 xpass=1 xfail=0 skip=2 error=2
+
+ : Let us finally make d.test pass.
+ echo : > d.extra
+ do_recheck --pass
+ test ! -r a.run
+ test ! -r b.run
+ test ! -r c.run
+ test -f d.run
+ count_test_results total=1 pass=0 fail=0 xpass=0 xfail=0 skip=1 error=0
+
+ : All tests have been successful or skipped, nothing should be re-run.
+ do_recheck --pass
+ test ! -r a.run
+ test ! -r b.run
+ test ! -r c.run
+ test ! -r d.run
+ count_test_results total=0 pass=0 fail=0 xpass=0 xfail=0 skip=0 error=0
+
+ cd $srcdir
+
+done
+
+:
diff --git a/tests/tap-skip-whole.test b/tests/tap-skip-whole.test
new file mode 100755
index 0000000..d3d08b8
--- /dev/null
+++ b/tests/tap-skip-whole.test
@@ -0,0 +1,79 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# Basic TAP test protocol support:
+# - special plan format to skip all the tests in a script.
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TEST_LOG_COMPILER = cat
+TESTS = foo.test err.test wget.test curl.test
+END
+
+cat > foo.test <<END
+1..0
+not ok 1
+not ok 2
+END
+
+cat > err.test <<END
+1..0$tab $tab
+not ok - xxx
+Bail out! some random failure
+END
+
+cat > wget.test <<END
+1..0 # wget(1) not installed
+not ok 1 # TODO not yet implemented
+ok 2 # SKIP who cares
+END
+
+cat > curl.test <<END
+1..0${tab}${tab} #${tab} ${tab}Can't connect to gnu.org!
+not ok foo
+ok bar # TODO but was supposed to fail
+END
+
+chmod a+x all.test
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+$MAKE check >stdout || { cat stdout; Exit 1; }
+cat stdout
+
+count_test_results total=4 pass=0 fail=0 xpass=0 xfail=0 skip=4 error=0
+
+grep '^SKIP: foo\.test$' stdout
+grep '^SKIP: bar\.test$' stdout
+grep '^SKIP: wget\.test (wget not installed)' stdout
+grep '^SKIP: curl\.test (Can'\''t connect to gnu\.org!)$' stdout
+
+:
diff --git a/tests/tap-summary.test b/tests/tap-summary.test
new file mode 100755
index 0000000..5cc36f4
--- /dev/null
+++ b/tests/tap-summary.test
@@ -0,0 +1,78 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support:
+# - testsuite summary
+# - colorized testsuite summary
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+# The following shell variables are influential for this function_
+# - expect_failure
+# - use_colors
+do_check ()
+{
+ cat > summary.exp
+ st=0
+ $MAKE check > stdout || st=$?
+ cat stdout
+ if $expect_failure = yes; then
+ test $st -gt 0 || Exit 1
+ else
+ test $st -eq 0 || Exit 1
+ fi
+ $PERL -w "$testsrcdir"/extract-testsuite-summary stdout > summary.got \
+ || fatal_ "cannot extract testsuite summary"
+ cat summary.exp
+ cat summary.got
+ if test $use_colors = yes; then
+ # Use cmp, not diff, because the files might contain binary data.
+ compare=cmp
+ else
+ compare=diff
+ fi
+ $compare summary.exp summary.got || Exit 1
+}
+
+br='============================================================================'
+
+cat > configure.in <<END
+AC_INIT([GNU AutoTAP], [5.12], address@hidden)
+AM_INIT_AUTOMAKE([parallel-tests])
+AC_SUBST([PERL], ['$PERL'])
+AC_CONFIG_FILES([Makefile])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TEST_LOG_COMPILER = cat
+TESTS = all.test
+END
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE --add-missing
+
+./configure
+
+Exit 1; # TODO
+
+:
diff --git a/tests/tap-todo-skip.test b/tests/tap-todo-skip.test
new file mode 100755
index 0000000..3bedf2a
--- /dev/null
+++ b/tests/tap-todo-skip.test
@@ -0,0 +1,185 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support:
+# - TODO and SKIP directives are case-insensitive
+# - TODO and SKIP directives can be followed optionally by a colon ":"
+# and by an optional explanation.
+# - our driver isn't fooled into recognizing TODO and SKIP directives
+# spuriously
+# - the reasons for TODO and SKIP, if present, are nicely printed in
+# the testsuite progress output
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TEST_LOG_COMPILER = cat
+TESTS = all.test
+END
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+# ----------------------------------------------------- #
+# Check all possible combinations of: #
+# - uppercase/lowercase #
+# - with/without colon character ":" #
+# - with/without explanatory message #
+# in TODO and SKIP directives. #
+# ----------------------------------------------------- #
+
+# There are 2^6 + 2^6 = 128 tests.
+echo 1..128 > all.test
+
+# These nested loops are clearer without indentation.
+
+for c1 in t T; do
+for c2 in o O; do
+for c3 in d D; do
+for c4 in o O; do
+for c5 in '' ':'; do
+for ex in '' 'foo'; do
+ echo "not ok # $c1$c2$c3$c4$c5$ex"
+done; done; done; done >> all.test
+
+for c1 in s S; do
+for c2 in k K; do
+for c3 in i I; do
+for c4 in p P; do
+for c5 in '' ':'; do
+for ex in '' 'foo'; do
+ echo "ok # $c1$c2$c3$c4$c5$ex"
+done; done; done; done >> all.test
+
+cat all.test # For debugging.
+
+$MAKE check >stdout || { cat stdout; Exit 1; }
+cat stdout
+
+count_test_results total=128 pass=0 fail=0 xpass=0 xfail=64 skip=64 error=0
+
+# -------------------------------------------------------- #
+# TODO ans SKIP directives aren't recognized spuriously. #
+# -------------------------------------------------------- #
+
+cat > all.test <<'END'
+1..9
+ok TODO
+ok - TODO
+ok 3 TODO
+ok 4 - TODO
+ok SKIP
+ok - SKIP
+ok 7 SKIP
+ok 8 - SKIP
+ok 9
+END
+
+$MAKE check >stdout || { cat stdout; Exit 1; }
+cat stdout
+
+count_test_results total=9 pass=9 fail=0 xpass=0 xfail=0 skip=0 error=0
+
+# --------------------------------------------------------------------- #
+# Some more ad-hoc checks, also verifying that the testsuite progress #
+# output on console behaves as expected. #
+# --------------------------------------------------------------------- #
+
+cat > stub.tap <<END
+1..9
+ok 1 # TODO
+ok 2 # TODO foo?
+ok 3 # TODO: bar!
+ok 4 aa # TODO
+ok 5 bb # TODO fnord 5
+ok 6 cc # TODO:${tab}fnord 6
+ok 7 - x # TODO
+ok 8 - y # TODO fnord $tab 8
+ok 9 - z # TODO: fnord 9 $tab
+END
+
+cat > stub.exp <<END
+: all.test 1
+: all.test 2 (foo?)
+: all.test 3 (bar!)
+: all.test 4 aa
+: all.test 5 bb (fnord 5)
+: all.test 6 cc (fnord 6)
+: all.test 7 - x
+: all.test 8 - y (fnord $tab 8)
+: all.test 9 - z (fnord 9 $tab)
+END
+
+# For "TODO" directives leading to XPASS results.
+
+cp stub.tap all.test
+sed 's/^/XPASS/' stub.exp > exp
+
+cat all.test
+$MAKE check >stdout && { cat stdout; Exit 1; }
+cat stdout
+count_test_results total=10 pass=0 fail=0 xpass=9 xfail=0 skip=0 error=0
+
+LC_ALL=C grep '^[A-Z][A-Z]*:' stdout > got
+cat exp
+cat got
+diff exp got
+
+# For "TODO" directives leading to XFAIL results.
+
+sed 's/^ok /not ok /' stub.tap > all.test
+sed 's/^/XFAIL/' stub.exp > exp
+
+cat all.test
+$MAKE check >stdout || { cat stdout; Exit 1; }
+cat stdout
+count_test_results total=10 pass=0 fail=0 xpass=0 xfail=9 skip=0 error=0
+
+LC_ALL=C grep '^[A-Z][A-Z]*:' stdout > got
+cat exp
+cat got
+diff exp got
+
+# For "SKIP" directives.
+
+sed 's/TODO/SKIP/' stub.tap > all.test
+sed 's/^/SKIP/' stub.exp > exp
+
+cat all.test
+$MAKE check >stdout || { cat stdout; Exit 1; }
+cat stdout
+count_test_results total=10 pass=0 fail=0 xpass=0 xfail=0 skip=9 error=0
+
+LC_ALL=C grep '^[A-Z][A-Z]*:' stdout > got
+cat exp
+cat got
+diff exp got
+
+
+:
diff --git a/tests/tap-with-and-without-number.test
b/tests/tap-with-and-without-number.test
new file mode 100755
index 0000000..f80a760
--- /dev/null
+++ b/tests/tap-with-and-without-number.test
@@ -0,0 +1,51 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support:
+# - numbered and unnumbered test results can coexist
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TESTS = all.test
+END
+
+cat > all.test <<'END'
+1..6
+ok 1 foo
+ok 2 - foo2
+ok - bar
+not ok baz # TODO
+not ok 4 - asd # TODO
+ok 5 # SKIP
+ok zardoz
+END
+
+$MAKE check >stdout || { cat stdout; Exit 1; }
+cat stdout
+count_test_results total=6 pass=4 fail=0 xpass=0 xfail=2 skip=1 error=0
+
+:
diff --git a/tests/tap-xfail-tests.test b/tests/tap-xfail-tests.test
new file mode 100755
index 0000000..f8c4e09
--- /dev/null
+++ b/tests/tap-xfail-tests.test
@@ -0,0 +1,70 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP and $(XFAIL_TESTS): test results without directives are turned
+# from PASS to XPASS and from FAIL to XFAIL.
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$testsrcdir"/tap-driver \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TEST_LOG_COMPILER = cat
+TESTS = the.test
+XFAIL_TESTS = $(TESTS)
+END
+
+cat > the.test <<END
+1..6
+ok 1
+not ok 2
+ok 3 # TODO
+not ok 4 # TODO
+ok 5 # SKIP
+not ok 6 # SKIP
+Bail out!
+END
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+$MAKE check >stdout && { cat stdout; Exit 1; }
+cat stdout
+
+count_test_results total=7 pass=0 fail=0 xpass=2 xfail=2 skip=2 error=1
+
+grep '^XPASS: the\.test 1$' stdout
+grep '^XFAIL: the\.test 2$' stdout
+grep '^XPASS: the\.test 3$' stdout
+grep '^XFAIL: the\.test 4$' stdout
+grep '^SKIP: the\.test 5$' stdout
+grep '^SKIP: the\.test 6$' stdout
+grep '^ERROR: the\.test - Bail out!$' stdout
+
+test `$FGREP -c ': the.test' stdout` -eq 7
+
+:
diff --git a/tests/test-driver-custom-multitest-recheck.test
b/tests/test-driver-custom-multitest-recheck.test
index 137ff96..8bbcc7c 100755
--- a/tests/test-driver-custom-multitest-recheck.test
+++ b/tests/test-driver-custom-multitest-recheck.test
@@ -20,6 +20,7 @@
# support, but also serves as a "usability test" for our APIs.
# See also related tests `test-driver-custom-multitest-recheck2.test'
# and `parallel-tests-recheck-override.test'.
+# Keep in sync with 'tap-recheck.test'.
parallel_tests=yes
. ./defs || Exit 1
--
1.7.2.3
- [GSoC] preliminary tests on TAP support,
Stefano Lattarini <=