[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[PATCH 5/5] {test-protocols} tap: add experimental TAP-aware driver
|
From: |
Stefano Lattarini |
|
Subject: |
[PATCH 5/5] {test-protocols} tap: add experimental TAP-aware driver |
|
Date: |
Fri, 15 Jul 2011 00:40:03 +0200 |
|
User-agent: |
KMail/1.13.3 (Linux/2.6.30-2-686; KDE/4.4.4; i686; ; ) |
* doc/automake.texi (Using the TAP test protocol): New section.
(Overview of Custom Test Drivers Support): Minor updates.
* lib/tap-driver: New script, TAP-aware test driver for Automake;
implemented in perl and based on TAP::Parser.
* lib/Makefile.am (dist_script_DATA): Add it.
* tests/tap-autonumber.test: New test.
* tests/tap-bailout.test: Likewise.
* tests/tap-basic.test: Likewise.
* tests/tap-deps.test: Likewise.
* tests/tap-diagnostic.test: Likewise.
* tests/tap-empty.test: Likewise.
* tests/tap-empty-diagnostic.test: Likewise.
* tests/tap-escape-directive.test: Likewise.
* tests/tap-exit.test: Likewise.
* tests/tap-fancy.test: Likewise.
* tests/tap-fancy2.test: Likewise.
* tests/tap-global-result.test: Likewise.
* tests/tap-html.test: Likewise.
* tests/tap-log.test: Likewise.
* tests/tap-merge-stdout-stderr.test: Likewise.
* tests/tap-more.test: Likewise.
* tests/tap-more2.test: Likewise.
* tests/tap-no-merge-stdout-stderr.test: Likewise.
* tests/tap-no-spurious-summary.test: Likewise.
* tests/tap-no-spurious.test: Likewise.
* tests/tap-not-ok-skip.test: Likewise.
* tests/tap-numeric-description.test: Likewise.
* tests/tap-out-of-order.test: Likewise.
* tests/tap-passthrough.test: Likewise.
* tests/tap-plan.test: Likewise.
* tests/tap-plan-errors.test: Likewise.
* tests/tap-plan-corner.test: Likewise.
* tests/tap-realtime.test: Likewise.
* tests/tap-recheck-logs.test: Likewise.
* tests/tap-recheck.test: Likewise.
* tests/tap-skip-whole.test: Likewise.
* tests/tap-summary.test: Likewise.
* tests/tap-todo-skip.test: Likewise.
* tests/tap-todo-skip-together.test: Likewise.
* tests/tap-todo-skip-whitespace.test: Likewise.
* tests/tap-skipall-whitespace.test: Likewise.
* tests/tap-unplanned.test: Likewise.
* tests/tap-whitespace-normalization.test: Likewise.
* tests/tap-with-and-without-number.test: Likewise.
* tests/tap-xfail-tests.test: Likewise.
* tests/tap-bad-prog.test: New xfailing test.
* tests/tap-color.test: Likewise.
* tests/tap-plan-corner2.test: Likewise.
* tests/tap-message-0.test: Likewise.
* tests/tap-signal.test: Likewise.
* tests/Makefile.am (TESTS, XFAIL_TESTS): Update.
---
ChangeLog | 55 ++++
doc/automake.texi | 94 ++++++-
lib/Makefile.am | 2 +-
lib/Makefile.in | 2 +-
lib/tap-driver | 363 +++++++++++++++++++++++
tests/Makefile.am | 51 ++++
tests/Makefile.in | 51 +++-
tests/tap-autonumber.test | 94 ++++++
tests/tap-bad-prog.test | 69 +++++
tests/tap-bailout.test | 155 ++++++++++
tests/tap-basic.test | 177 +++++++++++
tests/tap-color.test | 145 +++++++++
tests/tap-deps.test | 98 ++++++
tests/tap-diagnostic.test | 142 +++++++++
tests/tap-empty-diagnostic.test | 59 ++++
tests/tap-empty.test | 73 +++++
tests/tap-escape-directive.test | 57 ++++
tests/tap-exit.test | 70 +++++
tests/tap-fancy.test | 151 ++++++++++
tests/tap-fancy2.test | 147 +++++++++
tests/tap-global-result.test | 128 ++++++++
tests/tap-html.test | 93 ++++++
tests/tap-log.test | 174 +++++++++++
tests/tap-merge-stdout-stderr.test | 85 ++++++
tests/tap-message-0.test | 84 ++++++
tests/tap-more.test | 160 ++++++++++
tests/tap-more2.test | 93 ++++++
tests/tap-no-merge-stdout-stderr.test | 66 ++++
tests/tap-no-spurious-summary.test | 65 ++++
tests/tap-no-spurious.test | 114 +++++++
tests/tap-not-ok-skip.test | 57 ++++
tests/tap-numeric-description.test | 86 ++++++
tests/tap-out-of-order.test | 98 ++++++
tests/tap-passthrough-exit.test | 79 +++++
tests/tap-passthrough.test | 202 +++++++++++++
tests/tap-plan-corner.test | 81 +++++
tests/tap-plan-corner2.test | 67 +++++
tests/tap-plan-errors.test | 110 +++++++
tests/tap-plan.test | 69 +++++
tests/tap-realtime.test | 115 +++++++
tests/tap-recheck-logs.test | 132 ++++++++
tests/tap-recheck.test | 217 ++++++++++++++
tests/tap-signal.test | 65 ++++
tests/tap-skip-whole-whitespace.test | 68 +++++
tests/tap-skip-whole.test | 100 +++++++
tests/tap-summary.test | 363 +++++++++++++++++++++++
tests/tap-todo-skip-together.test | 57 ++++
tests/tap-todo-skip-whitespace.test | 118 ++++++++
tests/tap-todo-skip.test | 109 +++++++
tests/tap-unplanned.test | 177 +++++++++++
tests/tap-whitespace-normalization.test | 181 +++++++++++
tests/tap-with-and-without-number.test | 68 +++++
tests/tap-xfail-tests.test | 85 ++++++
tests/test-driver-custom-multitest-recheck.test | 1 +
54 files changed, 5812 insertions(+), 10 deletions(-)
create mode 100755 lib/tap-driver
create mode 100755 tests/tap-autonumber.test
create mode 100755 tests/tap-bad-prog.test
create mode 100755 tests/tap-bailout.test
create mode 100755 tests/tap-basic.test
create mode 100755 tests/tap-color.test
create mode 100755 tests/tap-deps.test
create mode 100755 tests/tap-diagnostic.test
create mode 100755 tests/tap-empty-diagnostic.test
create mode 100755 tests/tap-empty.test
create mode 100755 tests/tap-escape-directive.test
create mode 100755 tests/tap-exit.test
create mode 100755 tests/tap-fancy.test
create mode 100755 tests/tap-fancy2.test
create mode 100755 tests/tap-global-result.test
create mode 100755 tests/tap-html.test
create mode 100755 tests/tap-log.test
create mode 100755 tests/tap-merge-stdout-stderr.test
create mode 100755 tests/tap-message-0.test
create mode 100755 tests/tap-more.test
create mode 100755 tests/tap-more2.test
create mode 100755 tests/tap-no-merge-stdout-stderr.test
create mode 100755 tests/tap-no-spurious-summary.test
create mode 100755 tests/tap-no-spurious.test
create mode 100755 tests/tap-not-ok-skip.test
create mode 100755 tests/tap-numeric-description.test
create mode 100755 tests/tap-out-of-order.test
create mode 100755 tests/tap-passthrough-exit.test
create mode 100755 tests/tap-passthrough.test
create mode 100755 tests/tap-plan-corner.test
create mode 100755 tests/tap-plan-corner2.test
create mode 100755 tests/tap-plan-errors.test
create mode 100755 tests/tap-plan.test
create mode 100755 tests/tap-realtime.test
create mode 100755 tests/tap-recheck-logs.test
create mode 100755 tests/tap-recheck.test
create mode 100755 tests/tap-signal.test
create mode 100755 tests/tap-skip-whole-whitespace.test
create mode 100755 tests/tap-skip-whole.test
create mode 100755 tests/tap-summary.test
create mode 100755 tests/tap-todo-skip-together.test
create mode 100755 tests/tap-todo-skip-whitespace.test
create mode 100755 tests/tap-todo-skip.test
create mode 100755 tests/tap-unplanned.test
create mode 100755 tests/tap-whitespace-normalization.test
create mode 100755 tests/tap-with-and-without-number.test
create mode 100755 tests/tap-xfail-tests.test
diff --git a/ChangeLog b/ChangeLog
index d76b071..e81d0a4 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,58 @@
+2011-07-15 Stefano Lattarini <address@hidden>
+
+ tap: add experimental TAP-aware driver
+ * doc/automake.texi (Using the TAP test protocol): New section.
+ (Overview of Custom Test Drivers Support): Minor updates.
+ * lib/tap-driver: New script, TAP-aware test driver for Automake;
+ implemented in perl and based on TAP::Parser.
+ * lib/Makefile.am (dist_script_DATA): Add it.
+ * tests/tap-autonumber.test: New test.
+ * tests/tap-bailout.test: Likewise.
+ * tests/tap-basic.test: Likewise.
+ * tests/tap-deps.test: Likewise.
+ * tests/tap-diagnostic.test: Likewise.
+ * tests/tap-empty.test: Likewise.
+ * tests/tap-empty-diagnostic.test: Likewise.
+ * tests/tap-escape-directive.test: Likewise.
+ * tests/tap-exit.test: Likewise.
+ * tests/tap-fancy.test: Likewise.
+ * tests/tap-fancy2.test: Likewise.
+ * tests/tap-global-result.test: Likewise.
+ * tests/tap-html.test: Likewise.
+ * tests/tap-log.test: Likewise.
+ * tests/tap-merge-stdout-stderr.test: Likewise.
+ * tests/tap-more.test: Likewise.
+ * tests/tap-more2.test: Likewise.
+ * tests/tap-no-merge-stdout-stderr.test: Likewise.
+ * tests/tap-no-spurious-summary.test: Likewise.
+ * tests/tap-no-spurious.test: Likewise.
+ * tests/tap-not-ok-skip.test: Likewise.
+ * tests/tap-numeric-description.test: Likewise.
+ * tests/tap-out-of-order.test: Likewise.
+ * tests/tap-passthrough.test: Likewise.
+ * tests/tap-plan.test: Likewise.
+ * tests/tap-plan-errors.test: Likewise.
+ * tests/tap-plan-corner.test: Likewise.
+ * tests/tap-realtime.test: Likewise.
+ * tests/tap-recheck-logs.test: Likewise.
+ * tests/tap-recheck.test: Likewise.
+ * tests/tap-skip-whole.test: Likewise.
+ * tests/tap-summary.test: Likewise.
+ * tests/tap-todo-skip.test: Likewise.
+ * tests/tap-todo-skip-together.test: Likewise.
+ * tests/tap-todo-skip-whitespace.test: Likewise.
+ * tests/tap-skipall-whitespace.test: Likewise.
+ * tests/tap-unplanned.test: Likewise.
+ * tests/tap-whitespace-normalization.test: Likewise.
+ * tests/tap-with-and-without-number.test: Likewise.
+ * tests/tap-xfail-tests.test: Likewise.
+ * tests/tap-bad-prog.test: New xfailing test.
+ * tests/tap-color.test: Likewise.
+ * tests/tap-plan-corner2.test: Likewise.
+ * tests/tap-message-0.test: Likewise.
+ * tests/tap-signal.test: Likewise.
+ * tests/Makefile.am (TESTS, XFAIL_TESTS): Update.
+
2011-07-07 Stefano Lattarini <address@hidden>
tests defs: new auxiliary function 'count_test_results'
diff --git a/doc/automake.texi b/doc/automake.texi
index 16804cd..0c75a22 100644
--- a/doc/automake.texi
+++ b/doc/automake.texi
@@ -320,6 +320,7 @@ Support for test suites
* Simple Tests:: Listing programs and scripts in @code{TESTS}
* Simple Tests using parallel-tests:: More powerful test driver
* Custom Test Drivers:: Writing and using custom test drivers
+* Using the TAP test protocol:: Integrating test scripts that use the TAP
protocol
* DejaGnu Tests:: Interfacing with the external testing framework
* Install Tests:: Running tests on installed packages
@@ -8656,6 +8657,7 @@ are very similar.
* Simple Tests:: Listing programs and scripts in @code{TESTS}
* Simple Tests using parallel-tests:: More powerful test driver
* Custom Test Drivers:: Writing and using custom test drivers
+* Using the TAP test protocol:: Integrating test scripts that use the TAP
protocol
* DejaGnu Tests:: Interfacing with the external testing framework
* Install Tests:: Running tests on installed packages
@end menu
@@ -9037,12 +9039,9 @@ or work around the issue with inference rules to
generate the tests.
@subsection Overview of Custom Test Drivers Support
Starting form Automake version 1.12, the @option{parallel-tests} harness
-allows the package authors to use third-party custom test drivers, in case
-the
address@hidden FIXME: this should become "default ones" once we have TAP and
Subunit
-default one is inadequate for their purposes.
address@hidden FIXME: add this once we have TAP and Subunit
address@hidden "or do not support their testing protocol of choice."
+allows the package authors to use third-party custom test drivers, in
+case the default ones are inadequate for their purposes, or do not support
+their testing protocol of choice.
A custom test driver is expected to properly run the test programs
passed to it, including the command-line arguments passed to it,
@@ -9336,6 +9335,89 @@ is not the case, the HTML generation will not work,
although all the
other functionalities of the Automake testsuite harness should remain
untouched, and continue to work correctly.
address@hidden Using the TAP test protocol
address@hidden Using the TAP test protocol
+
+Brief introduction to TAP and its philosophy.
+
+Real-word uses of TAP: testsuite of @command{perl} and of many
+perl modules, and of @command{git}. Third-party libraries and
+utilities that can generate TAP (tell also to look at links
+below).
+
address@hidden
+A harness must only read TAP output from standard output and not
+from standard error. Lines written to standard output matching
address@hidden/^(not )?ok\b/} must be interpreted as test lines. All other
+lines must not be considered test output.
address@hidden quotation
+
address@hidden
+Here are some links to more extensive official or third-party documentation
+and resources:
address@hidden @bullet
address@hidden
address@hidden://search.cpan.org/~petdance/Test-Harness/lib/Test/Harness/TAP.pod,
+ @samp{Test::Harness::TAP}},
+the (mostly) official documentation about the TAP format and protocol.
address@hidden
address@hidden://search.cpan.org/~andya/Test-Harness/bin/prove,
+ @command{prove}},
+the most famous command-line TAP test driver, included in the distribution
+of @command{perl} and
address@hidden://search.cpan.org/~andya/Test-Harness/lib/Test/Harness.pm,
+ @samp{Test::Harness}}.
address@hidden
+The @uref{http://testanything.org/wiki/index.php/Main_Page,TAP wiki}.
address@hidden
+A ``gentle introduction'' to testing for perl coders:
address@hidden://search.cpan.org/dist/Test-Simple/lib/Test/Tutorial.pod,
+ @samp{Test::Tutorial}}.
address@hidden
address@hidden://search.cpan.org/~mschwern/Test-Simple/lib/Test/Simple.pm,
+ @samp{Test::Simple}}
+and
address@hidden://search.cpan.org/~mschwern/Test-Simple/lib/Test/More.pm,
+ @samp{Test::More}},
+the standard perl testing libraries, which are based on TAP.
address@hidden
address@hidden://www.eyrie.org/~eagle/software/c-tap-harness/,C TAP Harness},
+a C-based project implementing both a TAP producer and a TAP consumer.
address@hidden itemize
+
+Give example of output from @command{prove} command (when used in
+``verbose'' mode, with option @option{-v}), to which we'd like to
+remain somewhat compatible compatible, while also (and foremost)
+being compatible with the default @option{parallel-tests} driver.
+
+Example of output from out own TAP testsuite driver. Point out
+similarities with the output from @command{prove --verbose} and
+from the default @option{parallel-tests} driver.
+
+Option @option{--ignore-exit} causes the driver to ignore the exit
+status of the test scripts; by default, the driver will report an
+error if the script exit with status != 0.
address@hidden: also add a @option{--no-ignore-exit} option, for
+completeness?
+
+By default, TAP diagnostic (i.e., lines beginning with the @samp{#}
+character) are copied only in the @file{.log} file. The option
address@hidden causes the driver to display them in the testsuite
+progress output too (@emph{TODO}: give example). The option
address@hidden restore the default behaviour.
+
address@hidden
+Differences and incompatibilities with other TAP parsers and drivers:
address@hidden @bullet
address@hidden
+A @code{Bail out!} directive doesn't stop the whole testsuite, but only
+the test script it occurs into. This doesn't follows TAP specifications,
+but on the other hand maximize compatibility (and code sharing) with
+the ``hard error'' concept of the default @option{parallel-tests} driver.
address@hidden
address@hidden: there's surely something else ...
address@hidden itemize
+
@node DejaGnu Tests
@section DejaGnu Tests
diff --git a/lib/Makefile.am b/lib/Makefile.am
index a26d663..a9467f1 100644
--- a/lib/Makefile.am
+++ b/lib/Makefile.am
@@ -29,7 +29,7 @@ dist_pkgvdata_DATA = COPYING INSTALL texinfo.tex ansi2knr.c
ansi2knr.1 \
scriptdir = $(pkgvdatadir)
dist_script_DATA = config.guess config.sub install-sh mdate-sh missing \
mkinstalldirs elisp-comp ylwrap acinstall depcomp compile py-compile \
- symlink-tree ar-lib test-driver
+ symlink-tree ar-lib test-driver tap-driver
EXTRA_DIST = gnupload
diff --git a/lib/Makefile.in b/lib/Makefile.in
index 5f8aa91..b600402 100644
--- a/lib/Makefile.in
+++ b/lib/Makefile.in
@@ -242,7 +242,7 @@ dist_pkgvdata_DATA = COPYING INSTALL texinfo.tex ansi2knr.c
ansi2knr.1 \
scriptdir = $(pkgvdatadir)
dist_script_DATA = config.guess config.sub install-sh mdate-sh missing \
mkinstalldirs elisp-comp ylwrap acinstall depcomp compile py-compile \
- symlink-tree ar-lib test-driver
+ symlink-tree ar-lib test-driver tap-driver
EXTRA_DIST = gnupload
all: all-recursive
diff --git a/lib/tap-driver b/lib/tap-driver
new file mode 100755
index 0000000..acfcd59
--- /dev/null
+++ b/lib/tap-driver
@@ -0,0 +1,363 @@
+#! /usr/bin/env perl
+# Temporary/experimental TAP test driver for Automake.
+# TODO: should be rewritten portably (e.g., in awk or shell).
+
+# ---------------------------------- #
+# Imports, static data, and setup. #
+# ---------------------------------- #
+
+use warnings FATAL => 'all';
+use strict;
+use Getopt::Long ();
+use TAP::Parser;
+
+my $ME = "tap-driver";
+
+my $USAGE = <<'END';
+Usage:
+ tap-driver [--help|--version] --test-name=NAME --log-file=PATH
+ [--expect-failure={yes|no}] [--color-tests={yes|no}]
+ [--enable-hard-errors={yes|no}] [--merge|--no-merge]
+ [--ignore-exit] [--comments|--no-comments] [--] TEST-COMMAND
+The `--test-name' and `--log-file' options are mandatory.
+END
+
+my $HELP = "$ME: TAP-aware test driver for Automake testsuite harness." .
+ "\n" . $USAGE;
+
+my $VERSION = '(experimental version)';
+
+# ------------------- #
+# Global variables. #
+# ------------------- #
+
+my $lineno = 0; # Number of input lines seen so far.
+my $testno = 0; # Number of test results seen so far.
+my $plan_seen = 0; # Whether the TAP plan has been seen or not.
+my $parser; # TAP parser object (will be initialized later).
+
+# ----------------- #
+# Option parsing. #
+# ----------------- #
+
+my %cfg = (
+ "color-tests" => 0,
+ "expect-failure" => 0,
+ "enable-hard-errors" => 1,
+ "merge" => 0,
+ "comments" => 0,
+ "ignore-exit" => 0,
+);
+
+my $test_script_name = undef;
+my $log_file = undef;
+
+Getopt::Long::GetOptions (
+ 'help' => sub { print $HELP; exit 0; },
+ 'version' => sub { print "$ME $VERSION"; exit 0; },
+ 'test-name=s' => \$test_script_name,
+ 'log-file=s' => \$log_file,
+ 'color-tests=s' => \&bool_opt,
+ 'expect-failure=s' => \&bool_opt,
+ 'enable-hard-errors=s' => \&bool_opt,
+ 'comments' => sub { $cfg{"comments"} = 1; },
+ 'no-comments' => sub { $cfg{"comments"} = 0; },
+ 'merge' => sub { $cfg{"merge"} = 1; },
+ 'no-merge' => sub { $cfg{"merge"} = 0; },
+ 'ignore-exit' => sub { $cfg{"ignore-exit"} = 1; },
+ ) or exit 1;
+
+# -------------- #
+# Subroutines. #
+# -------------- #
+
+sub bool_opt ($$)
+{
+ my ($opt, $val) = @_;
+ if ($val =~ /^(?:y|yes)\z/i)
+ {
+ $cfg{$opt} = 1;
+ }
+ elsif ($val =~ /^(?:n|no)\z/i)
+ {
+ $cfg{$opt} = 0;
+ }
+ else
+ {
+ die "invalid argument '$val' for option '$opt'\n";
+ }
+}
+
+TAP_PEEKING :
+{
+ my @tap_lines = ();
+
+ sub get_tap_line ()
+ {
+ @tap_lines > 0 ? pop @tap_lines : $parser->next;
+ }
+
+ sub unget_tap_line ($)
+ {
+ push @tap_lines, @_;
+ }
+
+ sub peek_tap_line ()
+ {
+ my $res = get_tap_line;
+ unget_tap_line ($res);
+ return $res;
+ }
+
+}
+
+TEST_RESULTS :
+{
+ my (@test_results, %test_results);
+
+ sub add_test_result ($)
+ {
+ my $res = shift;
+ push @test_results, $res;
+ $test_results{$res} = 1;
+ }
+
+ sub get_test_results ()
+ {
+ return @test_results;
+ }
+
+ # FIXME: this can certainly be improved ...
+ sub get_global_test_result ()
+ {
+ my @results = keys %test_results;
+ return "ERROR" if exists $test_results{"ERROR"};
+ return "SKIP" if @results == 1 && $results[0] eq "SKIP";
+ return "FAIL" if exists $test_results{"FAIL"};
+ return "FAIL" if exists $test_results{"XPASS"};
+ return "PASS";
+ }
+
+}
+
+sub start (@)
+{
+ # Redirect stderr and stdout to a temporary log file. Save the
+ # original stdout stream, since we need it to print testsuite
+ # progress output.
+ open OLDOUT, ">&STDOUT" or die "duplicating stdout: $!\n";
+ open TMP, ">$log_file-t" or die "opening $log_file-t: $!\n";
+ open STDOUT, ">&TMP" or die "redirecting stdout: $!\n";
+ open STDERR, ">&TMP" or die "redirecting stderr: $!\n";
+ $parser = TAP::Parser->new ({ exec => address@hidden, merge => $cfg{merge}
});
+ $parser->ignore_exit(1) if $cfg{"ignore-exit"};
+}
+
+sub finish ()
+{
+ open LOG, ">", $log_file or die "opening $log_file: $!\n";
+
+ # We need to declare a global test result in order to have
+ # "make recheck" working.
+ my $global_result = get_global_test_result;
+ my $global_result_line = "$global_result: $test_script_name";
+ print LOG "$global_result_line\n";
+ print LOG "=" x length ($global_result_line) . "\n";
+ print LOG "\n";
+
+ foreach (get_test_results)
+ {
+ print LOG ":test-result: $_\n";
+ # Add extra trailing empty lines to support reStructuredText -> HTML
+ # conversion.
+ print LOG "\n";
+ }
+
+ # So that the output from the test script won't be parsed for test
+ # results (which would potentially expose us to false positives).
+ print LOG ":test-result: END\n";
+ print LOG "\n";
+
+ close TMP or die "closing $log_file-t: $!\n";
+ # FIXME: remove this hack!
+ my $test_output = `cat $log_file-t && rm -f $log_file-t`;
+ print LOG $test_output;
+ close LOG or die "closing $log_file: $!\n";
+
+ exit 0;
+}
+
+sub stringify_test_result ($)
+{
+ my $result = shift;
+ my $PASS = $cfg{"expect-failure"} ? "XPASS": "PASS";
+ my $FAIL = $cfg{"expect-failure"} ? "XFAIL": "FAIL";
+ if ($result->is_unplanned || $result->number != $testno)
+ {
+ return "ERROR";
+ }
+ elsif (!$result->directive)
+ {
+ return $result->is_ok ? $PASS: $FAIL;
+ }
+ elsif ($result->has_todo)
+ {
+ return $result->is_actual_ok ? "XPASS" : "XFAIL";
+ }
+ elsif ($result->has_skip)
+ {
+ return $result->is_ok ? "SKIP" : $FAIL;
+ }
+ die "INTERNAL ERROR"; # NOTREACHED
+}
+
+sub console_output (@)
+{
+ return unless @_ > 0;
+ my $msg = join ("\n", @_) . "\n";
+ print OLDOUT $msg;
+ # Log the result in the log file too, to help debugging (this is
+ # especially true when said result is a TAP error or "Bail out!").
+ print $msg;
+}
+
+sub testuite_error ($)
+{
+ add_test_result "ERROR";
+ console_output "ERROR: $test_script_name - $_[0]";
+}
+
+sub handle_tap_test ($)
+{
+ $testno++;
+ my $test = shift;
+
+ my $test_result = stringify_test_result $test;
+ my $string = "$test_result: $test_script_name " . $test->number;
+
+ if (my $description = $test->description)
+ {
+ $string .= " $description";
+ }
+ if ($test->is_unplanned)
+ {
+ $string .= " # UNPLANNED";
+ }
+ elsif ($test->number != $testno)
+ {
+ $string .= " # OUT-OF-ORDER (expecting $testno)";
+ }
+ elsif (my $directive = $test->directive)
+ {
+ $string .= " # $directive";
+ if (my $explanation = $test->explanation)
+ {
+ $string .= " $explanation";
+ }
+ }
+ add_test_result $test_result;
+ console_output $string;
+}
+
+sub handle_tap_plan ($)
+{
+ my $plan = shift;
+ testuite_error "multiple test plans" if $plan_seen;
+ $plan_seen = 1;
+ # TAP plan must be either in the first or in the last line.
+ if ($lineno > 1 && peek_tap_line)
+ {
+ testuite_error "test plan in middle of output";
+ return;
+ }
+ # Nothing more to do, unless the plan contains a SKIP directive.
+ return
+ if not defined $plan->directive && length ($plan->directive) > 0;
+ my $string = "SKIP: $test_script_name";
+ if (my $explanation = $plan->explanation)
+ {
+ $string .= " - $explanation";
+ }
+ add_test_result "SKIP";
+ console_output $string;
+ finish;
+}
+
+sub handle_tap_bailout ($)
+{
+ my ($bailout, $msg) = ($_[0], "Bail out!");
+ $msg .= " " . $bailout->explanation if $bailout->explanation;
+ testuite_error $msg;
+ finish;
+}
+
+sub handle_tap_comment ($)
+{
+ return unless $cfg{comments};
+ my $comment = $_[0]->comment;
+ console_output "# $test_script_name: $comment"
+ if length $comment;
+}
+
+sub main (@)
+{
+ start @_;
+
+ while (defined (my $cur = get_tap_line))
+ {
+ # Verbatim copy any input line into the log file.
+ print $cur->raw . "\n";
+ $lineno++;
+ if ($cur->is_plan)
+ {
+ handle_tap_plan ($cur);
+ }
+ elsif ($cur->is_test)
+ {
+ handle_tap_test ($cur);
+ }
+ elsif ($cur->is_comment)
+ {
+ handle_tap_comment ($cur);
+ }
+ elsif ($cur->is_bailout)
+ {
+ handle_tap_bailout ($cur);
+ }
+ }
+ if (!$plan_seen)
+ {
+ testuite_error "missing test plan";
+ }
+ elsif ($parser->tests_planned != $parser->tests_run)
+ {
+ my ($planned, $run) = ($parser->tests_planned, $parser->tests_run);
+ my $bad_amount = $run > $planned ? "many" : "few";
+ testuite_error (sprintf "too %s tests run (expected %d, got %d)",
+ $bad_amount, $planned, $run);
+ }
+ # TODO: we should probably use $parser->wait here, to catch signals too
+ if ($parser->exit != 0)
+ {
+ my $msg = sprintf "exited with status %d", $parser->exit;
+ if ($cfg{"ignore-exit"})
+ {
+ # Log the exit status of the script anyway, even if it is not
+ # considered to be an error, to help debugging.
+ print "INFO: $test_script_name - $msg\n";
+ }
+ else
+ {
+ testuite_error $msg;;
+ }
+ }
+ finish;
+}
+
+# ----------- #
+# Main code. #
+# ----------- #
+
+main @ARGV;
+
+# vim: ft=perl ts=4 sw=4 et
diff --git a/tests/Makefile.am b/tests/Makefile.am
index 0d41a1c..2ef4d70 100644
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -27,6 +27,11 @@ gcj6.test \
override-conditional-2.test \
pr8365-remake-timing.test \
yacc-dist-nobuild-subdir.test \
+tap-bad-prog.test \
+tap-color.test \
+tap-plan-corner2.test \
+tap-message-0.test \
+tap-signal.test \
txinfo5.test
@@ -980,6 +985,52 @@ suffix13.test \
symlink.test \
symlink2.test \
syntax.test \
+tap-autonumber.test \
+tap-bad-prog.test \
+tap-bailout.test \
+tap-basic.test \
+tap-color.test \
+tap-deps.test \
+tap-diagnostic.test \
+tap-empty.test \
+tap-empty-diagnostic.test \
+tap-escape-directive.test \
+tap-exit.test \
+tap-signal.test \
+tap-fancy.test \
+tap-fancy2.test \
+tap-global-result.test \
+tap-html.test \
+tap-log.test \
+tap-more.test \
+tap-more2.test \
+tap-message-0.test \
+tap-merge-stdout-stderr.test \
+tap-no-merge-stdout-stderr.test \
+tap-no-spurious-summary.test \
+tap-no-spurious.test \
+tap-not-ok-skip.test \
+tap-numeric-description.test \
+tap-out-of-order.test \
+tap-passthrough.test \
+tap-passthrough-exit.test \
+tap-plan.test \
+tap-plan-errors.test \
+tap-plan-corner.test \
+tap-plan-corner2.test \
+tap-realtime.test \
+tap-recheck-logs.test \
+tap-recheck.test \
+tap-summary.test \
+tap-skip-whole.test \
+tap-skip-whole-whitespace.test \
+tap-todo-skip.test \
+tap-todo-skip-whitespace.test \
+tap-todo-skip-together.test \
+tap-unplanned.test \
+tap-whitespace-normalization.test \
+tap-with-and-without-number.test \
+tap-xfail-tests.test \
tags.test \
tags2.test \
tagsub.test \
diff --git a/tests/Makefile.in b/tests/Makefile.in
index c72b1cf..c5e1009 100644
--- a/tests/Makefile.in
+++ b/tests/Makefile.in
@@ -273,8 +273,9 @@ EXTRA_DIST = ChangeLog-old gen-parallel-tests
instspc-tests.sh \
extract-testsuite-summary
XFAIL_TESTS = all.test auxdir2.test cond17.test gcj6.test \
override-conditional-2.test pr8365-remake-timing.test \
- yacc-dist-nobuild-subdir.test txinfo5.test \
- $(instspc_xfail_tests)
+ yacc-dist-nobuild-subdir.test tap-bad-prog.test tap-color.test \
+ tap-plan-corner2.test tap-message-0.test tap-signal.test \
+ txinfo5.test $(instspc_xfail_tests)
parallel_tests = backcompat5-p.test check-exported-srcdir-p.test \
check-tests-in-builddir-p.test check-tests_environment-p.test \
check-p.test check11-p.test check12-p.test check2-p.test \
@@ -1227,6 +1228,52 @@ suffix13.test \
symlink.test \
symlink2.test \
syntax.test \
+tap-autonumber.test \
+tap-bad-prog.test \
+tap-bailout.test \
+tap-basic.test \
+tap-color.test \
+tap-deps.test \
+tap-diagnostic.test \
+tap-empty.test \
+tap-empty-diagnostic.test \
+tap-escape-directive.test \
+tap-exit.test \
+tap-signal.test \
+tap-fancy.test \
+tap-fancy2.test \
+tap-global-result.test \
+tap-html.test \
+tap-log.test \
+tap-more.test \
+tap-more2.test \
+tap-message-0.test \
+tap-merge-stdout-stderr.test \
+tap-no-merge-stdout-stderr.test \
+tap-no-spurious-summary.test \
+tap-no-spurious.test \
+tap-not-ok-skip.test \
+tap-numeric-description.test \
+tap-out-of-order.test \
+tap-passthrough.test \
+tap-passthrough-exit.test \
+tap-plan.test \
+tap-plan-errors.test \
+tap-plan-corner.test \
+tap-plan-corner2.test \
+tap-realtime.test \
+tap-recheck-logs.test \
+tap-recheck.test \
+tap-summary.test \
+tap-skip-whole.test \
+tap-skip-whole-whitespace.test \
+tap-todo-skip.test \
+tap-todo-skip-whitespace.test \
+tap-todo-skip-together.test \
+tap-unplanned.test \
+tap-whitespace-normalization.test \
+tap-with-and-without-number.test \
+tap-xfail-tests.test \
tags.test \
tags2.test \
tagsub.test \
diff --git a/tests/tap-autonumber.test b/tests/tap-autonumber.test
new file mode 100755
index 0000000..745a639
--- /dev/null
+++ b/tests/tap-autonumber.test
@@ -0,0 +1,94 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support:
+# - unnumbered tests are OK, as long as their final total number
+# agrees with the plan
+# - test results without number get automatically numbered in the
+# console progress output
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TEST_LOG_COMPILER = cat
+TESTS = all.test
+EXTRA_DIST = $(TESTS)
+END
+
+cat > all.test <<'END'
+1..14
+ok
+ok foo
+ok - foo2
+not ok
+not ok bar
+not ok - bar2
+; See that we can intermingle different kind of results without
+; messing up the autonumbering
+ok
+ok # TODO
+not ok # TODO who cares?
+ok
+not ok
+ok muuu # SKIP
+not ok
+ok
+END
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+$MAKE check >stdout && { cat stdout; Exit 1; }
+cat stdout
+count_test_results total=14 pass=6 fail=5 xpass=1 xfail=1 skip=1 error=0
+
+cat > exp <<'END'
+PASS: all.test 1
+PASS: all.test 2 foo
+PASS: all.test 3 - foo2
+FAIL: all.test 4
+FAIL: all.test 5 bar
+FAIL: all.test 6 - bar2
+PASS: all.test 7
+XPASS: all.test 8 # TODO
+XFAIL: all.test 9 # TODO who cares?
+PASS: all.test 10
+FAIL: all.test 11
+SKIP: all.test 12 muuu # SKIP
+FAIL: all.test 13
+PASS: all.test 14
+END
+
+$FGREP ': all.test' stdout > got
+
+cat exp
+cat got
+diff exp got
+
+:
diff --git a/tests/tap-bad-prog.test b/tests/tap-bad-prog.test
new file mode 100755
index 0000000..af54b2e
--- /dev/null
+++ b/tests/tap-bad-prog.test
@@ -0,0 +1,69 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support:
+# - missing, unreadable, or not-executable test scripts cause proper
+# error reports
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TESTS = none.test noread.test noexec.test
+none.test:
+END
+
+cat > noexec.test <<'END'
+#!/bin/sh
+echo 1..1
+ok 1
+END
+
+./noexec.test && skip_ "can't have non-executable files"
+
+cp noexec.test noread.test
+chmod a-r noread
+
+if test -r noread.test || cat noread.test; then
+ skip_ "can't have readonly files"
+fi
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+$MAKE check >stdout && { cat stdout; Exit 1; }
+cat stdout
+
+# FIXME: maybe better error messages?
+grep '^ERROR: none.test' stdout
+grep '^ERROR: noexec.test' stdout
+grep '^ERROR: noread.test' stdout
+
+count_test_results total=3 pass=0 fail=0 xpass=0 xfail=0 skip=0 error=3
+
+:
diff --git a/tests/tap-bailout.test b/tests/tap-bailout.test
new file mode 100755
index 0000000..6c0a588
--- /dev/null
+++ b/tests/tap-bailout.test
@@ -0,0 +1,155 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# Basic TAP test protocol support:
+# - "Bail out!" magic
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TEST_LOG_COMPILER = cat
+TESTS = a.test b.test c.test d.test e.test
+END
+
+: > exp
+
+#------------------------------------------------------------------
+
+# Bailout without explanation.
+
+cat > a.test <<END
+1..4
+ok 1
+not ok 2
+Bail out!
+not ok 3
+ok 4 # SKIP
+END
+
+cat >> exp <<END
+PASS: a.test 1
+FAIL: a.test 2
+ERROR: a.test - Bail out!
+END
+
+# pass += 1, fail +=1, error += 1
+
+#------------------------------------------------------------------
+
+# Bailout with explanation.
+
+cat > b.test <<END
+1..7
+ok 1 # SKIP
+ok 2 # TODO
+not ok 3 # TODO
+Bail out! We're out of disk space.
+ok 4
+not ok 5
+not ok 6 # TODO
+ok 7 # TODO
+END
+
+cat >> exp <<END
+SKIP: b.test 1 # SKIP
+XPASS: b.test 2 # TODO
+XFAIL: b.test 3 # TODO
+ERROR: b.test - Bail out! We're out of disk space.
+END
+
+# skip += 1, xpass += 1, xfail += 1, error += 1
+
+#------------------------------------------------------------------
+
+# Bail out before the test plan.
+
+cat > c.test <<END
+ok 1
+ok 2
+Bail out! BOOOH!
+1..2
+END
+
+cat >> exp <<END
+PASS: c.test 1
+PASS: c.test 2
+ERROR: c.test - Bail out! BOOOH!
+END
+
+# pass += 2, error += 1
+
+#------------------------------------------------------------------
+
+# Bailout on the first line.
+
+cat > d.test <<END
+Bail out! mktemp -d: Permission denied
+ok 1
+END
+
+echo 'ERROR: d.test - Bail out! mktemp -d: Permission denied' >> exp
+
+# error += 1
+
+#------------------------------------------------------------------
+
+# TAP input comprised only of a bailout directive.
+
+cat > e.test <<END
+Bail out!
+END
+
+echo "ERROR: e.test - Bail out!" >> exp
+
+# error += 1
+
+#------------------------------------------------------------------
+
+# Doing the sums above, we have:
+test_counts='total=12 pass=3 fail=1 xpass=1 xfail=1 skip=1 error=5'
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+$MAKE check >stdout && { cat stdout; Exit 1; }
+cat stdout
+
+count_test_results $test_counts
+
+LC_ALL=C sort exp > t
+mv -f t exp
+
+# We need the sort below to account for parallel make usage.
+grep ': [abcde]\.test' stdout | LC_ALL=C sort > got
+
+cat exp
+cat got
+diff exp got
+
+:
diff --git a/tests/tap-basic.test b/tests/tap-basic.test
new file mode 100755
index 0000000..26a749d
--- /dev/null
+++ b/tests/tap-basic.test
@@ -0,0 +1,177 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# Basic TAP support:
+# - LOG_COMPILER support;
+# - basic support for TODO and SKIP directives, and "Bail out!" magic;
+# - testsuite progress output on console;
+# - runtime overriding of TESTS and TEST_LOGS;
+# - correct counts of test results (both in summary and in progress
+# output on console).
+# Note that some of the features checked here are checked in other
+# test cases too, usually in a more thorough and detailed way.
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+## Defining LOG_COMPILER should work and not intefere with the
+## tap-driver script.
+TEST_LOG_COMPILER = cat
+TESTS = success.test
+
+ok.test:
+ echo '1..3' > address@hidden
+ echo 'ok 1' >> address@hidden
+ echo 'not ok 2 # TODO' >>address@hidden
+ echo 'ok 3 # SKIP' >>address@hidden
+ cat address@hidden ;: For debugging.
+ mv -f address@hidden $@
+END
+
+cat > success.test << 'END'
+1..20
+ok 1
+ok 2 two
+ok 3 - three
+ok 4 four four
+not ok 5
+not ok 6 six
+not ok 7 - seven
+not ok 8 eight eight
+ok 9 # TODO
+ok 10 ten # TODO
+ok 11 - eleven # TODO
+ok 12 twelve twelve # TODO
+not ok 13 # TODO
+not ok 14 fourteen # TODO
+not ok 15 - fifteen # TODO
+not ok 16 sixteen sixteen # TODO
+ok 17 # SKIP
+ok 18 eighteen # SKIP
+ok 19 - nineteen # SKIP
+ok 20 twenty twenty # SKIP
+END
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+# Basilar usage and testsuite progress output.
+
+$MAKE check >stdout && { cat stdout; Exit 1; }
+cat stdout
+
+count_test_results total=20 pass=4 fail=4 xpass=4 xfail=4 skip=4 error=0
+
+test -f success.log
+test -f test-suite.log
+
+cat > exp << 'END'
+PASS: success.test 1
+PASS: success.test 2 two
+PASS: success.test 3 - three
+PASS: success.test 4 four four
+FAIL: success.test 5
+FAIL: success.test 6 six
+FAIL: success.test 7 - seven
+FAIL: success.test 8 eight eight
+XPASS: success.test 9 # TODO
+XPASS: success.test 10 ten # TODO
+XPASS: success.test 11 - eleven # TODO
+XPASS: success.test 12 twelve twelve # TODO
+XFAIL: success.test 13 # TODO
+XFAIL: success.test 14 fourteen # TODO
+XFAIL: success.test 15 - fifteen # TODO
+XFAIL: success.test 16 sixteen sixteen # TODO
+SKIP: success.test 17 # SKIP
+SKIP: success.test 18 eighteen # SKIP
+SKIP: success.test 19 - nineteen # SKIP
+SKIP: success.test 20 twenty twenty # SKIP
+END
+
+$FGREP ': success.test' stdout > got
+
+cat exp
+cat got
+diff exp got
+
+# Override TESTS from the command line.
+
+rm -f *.log *.test
+
+cat > bail.test <<'END'
+1..1
+Bail out!
+ok 1
+END
+
+TESTS=bail.test $MAKE -e check >stdout && { cat stdout; Exit 1; }
+cat stdout
+
+count_test_results total=1 pass=0 fail=0 xpass=0 xfail=0 skip=0 error=1
+
+test ! -f success.log
+test -f bail.log
+test -f test-suite.log
+
+grep '^ERROR: bail\.test - Bail out!' stdout
+grep '^PASS:' stdout && Exit 1
+test `$FGREP -c ': bail.test' stdout` -eq 1
+$FGREP 'success.test' stdout && Exit 1
+
+# Override TEST_LOGS from the command line, making it point to a test
+# (ok.test) that have to be generated at make time.
+
+rm -f *.log *.test
+
+TEST_LOGS=ok.log $MAKE -e check >stdout || { cat stdout; Exit 1; }
+cat stdout
+
+count_test_results total=3 pass=1 fail=0 xpass=0 xfail=1 skip=1 error=0
+
+test -f ok.test
+test -f ok.log
+test ! -f success.log
+test ! -f bail.log
+test -f test-suite.log
+
+$EGREP '(bail|success)\.test' stdout && Exit 1
+
+cat > exp << 'END'
+PASS: ok.test 1
+XFAIL: ok.test 2 # TODO
+SKIP: ok.test 3 # SKIP
+END
+
+$FGREP ': ok.test' stdout > got
+
+cat exp
+cat got
+diff exp got
+
+:
diff --git a/tests/tap-color.test b/tests/tap-color.test
new file mode 100755
index 0000000..8ed7265
--- /dev/null
+++ b/tests/tap-color.test
@@ -0,0 +1,145 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support:
+# - colorization of TAP results and diagnostic messages
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+TERM=ansi; export TERM
+
+esc=''
+# Escape `[' for grep, below.
+red="$esc\[0;31m"
+grn="$esc\[0;32m"
+lgn="$esc\[1;32m"
+blu="$esc\[1;34m"
+mgn="$esc\[0;35m"
+std="$esc\[m"
+
+# Check that grep can parse nonprinting characters.
+# BSD 'grep' works from a pipe, but not a seekable file.
+# GNU or BSD 'grep -a' works on files, but is not portable.
+case `echo "$std" | grep .` in
+ "$std") ;;
+ *) echo "$me: grep can't parse nonprinting characters" >&2; Exit 77;;
+esac
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+AUTOMAKE_OPTIONS = color-tests
+AM_TEST_LOG_DRIVER_FLAGS = --comments
+TEST_LOG_COMPILER = cat
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TESTS = all.test skip.test bailout.test plan-errors.test
+END
+
+cat > all.test << 'END'
+1..5
+ok 1 - foo
+# Hi! I shouldn't be colorized!
+not ok 2 - bar # TODO td
+ok 3 - baz # SKIP sk
+not ok 4 - quux
+ok 5 - zardoz # TODO
+END
+
+cat > skip.test << 'END'
+1..0 # Whole script not run
+END
+
+cat > errors.test << 'END'
+1..1
+ok 1
+Bail out!
+END
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+AM_COLOR_TESTS=always $MAKE check >stdout && { cat stdout; Exit 1; }
+cat stdout
+
+test_color ()
+{
+ # Not a useless use of cat; see above comments about grep.
+ cat stdout | grep "^${grn}PASS${std}: all\.test 1 - foo$"
+ cat stdout | grep "^${lgn}XFAIL${std}: all\.test 2 - bar # TODO td$"
+ cat stdout | grep "^${blu}SKIP${std}: all\.test 3 - baz # SKIP sk$"
+ cat stdout | grep "^${red}FAIL${std}: all\.test 4 - quux$"
+ cat stdout | grep "^${red}XPASS${std}: all\.test 5 - zardoz # TODO$"
+ cat stdout | grep "^${blu}SKIP${std}: skip\.test # Whole script not run$"
+ cat stdout | grep "^${grn}PASS${std}: bailout\.test 1$"
+ cat stdout | grep "^${mgn}ERROR${std}: all\.test - Bail out!$"
+ # Diagnostic messages shouldn't be colorized.
+ cat stdout | grep "^# all\.test: Hi! I shouldn't be colorized!$"
+ :
+}
+
+test_no_color ()
+{
+ # With make implementations that, like Solaris make, in case of errors
+ # print the whole failing recipe on standard output, we should content
+ # ourselves with a laxer check, to avoid false positives.
+ # Keep this in sync with lib/am/check.am:$(am__color_tests).
+ if $FGREP '= Xalways || test -t 1 ' stdout; then
+ # Extra verbose make, resort to laxer checks.
+ # But we also want to check that the testsuite summary is not unduly
+ # colorized.
+ (
+ set +e # In case some grepped regex below isn't matched.
+ # Not a useless use of cat; see above comments about grep.
+ cat stdout | grep "TOTAL.*:"
+ cat stdout | grep "PASS.*:"
+ cat stdout | grep "FAIL.*:"
+ cat stdout | grep "SKIP.*:"
+ cat stdout | grep "XFAIL.*:"
+ cat stdout | grep "XPASS.*:"
+ cat stdout | grep "ERROR.*:"
+ cat stdout | grep "^#"
+ cat stdout | grep 'test.*expected'
+ cat stdout | grep 'test.*not run'
+ cat stdout | grep '===='
+ cat stdout | grep '[Ss]ee .*test-suite\.log'
+ cat stdout | grep '[Tt]estsuite summary'
+ ) | grep "$esc" && Exit 1
+ : For shells with broken 'set -e'
+ else
+ cat stdout | grep "$esc" && Exit 1
+ : For shells with broken 'set -e'
+ fi
+}
+
+AM_COLOR_TESTS=always $MAKE check >stdout && { cat stdout; Exit 1; }
+cat stdout
+test_color
+
+$MAKE -e check >stdout && { cat stdout; Exit 1; }
+cat stdout
+test_no_color
+
+:
diff --git a/tests/tap-deps.test b/tests/tap-deps.test
new file mode 100755
index 0000000..ec74485
--- /dev/null
+++ b/tests/tap-deps.test
@@ -0,0 +1,98 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# Basic TAP test protocol support:
+# - dependencies between test scripts
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+# The tests are *deliberately* listed in inversed order here.
+TESTS = c.test b.test a.test
+b.log: a.log
+c.log: b.log
+END
+
+cat > a.test << 'END'
+#!/bin/sh
+echo 1..2
+echo ok 1
+sleep '3' # FIXME: quotes to please maintainer-check
+echo ok 2
+: > a.run
+END
+
+cat > b.test << 'END'
+#!/bin/sh
+echo 1..2
+if test -f a.run; then
+ echo ok 1
+else
+ echo not ok 1
+fi
+sleep '3' # FIXME: quotes to please maintainer-check
+echo ok 2
+: > b.run
+END
+
+cat > c.test << 'END'
+#!/bin/sh
+echo 1..1
+test -f b.run || { echo 'Bail out!'; exit 1; }
+echo ok 1
+rm -f a.run b.run
+END
+
+chmod a+x *.test
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+$MAKE check >stdout || { cat stdout; Exit 1; }
+cat stdout
+
+count_test_results total=5 pass=5 fail=0 xpass=0 xfail=0 skip=0 error=0
+
+cat > exp << 'END'
+PASS: a.test 1
+PASS: a.test 2
+PASS: b.test 1
+PASS: b.test 2
+PASS: c.test 1
+END
+
+grep ': [abc]\.test' stdout > got
+
+cat exp
+cat got
+diff exp got
+
+# TODO: it would be nice to also redo the checks forcing parallel make...
+
+:
diff --git a/tests/tap-diagnostic.test b/tests/tap-diagnostic.test
new file mode 100755
index 0000000..caef651
--- /dev/null
+++ b/tests/tap-diagnostic.test
@@ -0,0 +1,142 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support:
+# - diagnostic messages (TAP lines with leading "#")
+# - flags '--comments' and '--no-comments' of the TAP test driver
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TEST_LOG_COMPILER = cat
+TESTS = all.test
+END
+
+metacharacters=\''"\$!&()[]<>#;^?*'
+
+cat > all.test <<END
+1..4
+# Hi! I'm a comment.
+# Tests begin.
+ok 1
+not ok 2 - foo # TODO
+ok 3 - bar # SKIP
+# Tests end.
+ok - zardoz
+# Shell metacharacters here: $metacharacters
+.# Leading characters before "#", not a TAP diagnostic line.
+x # Leading characters before "#", not a TAP diagnostic line.
+ # Leading whitespace before "#", not a TAP diagnostic line.
+${tab}# Leading whitespace before "#", not a TAP diagnostic line.
+ ${tab} # Leading whitespace before "#", not a TAP diagnostic line.
+END
+
+cat > exp <<END
+# all.test: Hi! I'm a comment.
+# all.test: Tests begin.
+PASS: all.test 1
+XFAIL: all.test 2 - foo # TODO
+SKIP: all.test 3 - bar # SKIP
+# all.test: Tests end.
+PASS: all.test 4 - zardoz
+# all.test: Shell metacharacters here: $metacharacters
+END
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+$MAKE check >stdout || { cat stdout; Exit 1; }
+cat stdout
+$EGREP -i "#.*all\\.test|a comment|(Tests|Shell) " stdout && Exit 1
+count_test_results total=4 pass=2 fail=0 xpass=0 xfail=1 skip=1 error=0
+
+echo 'AM_TEST_LOG_DRIVER_FLAGS = --comments' >> Makefile
+$MAKE check >stdout || { cat stdout; Exit 1; }
+cat stdout
+$FGREP ' all.test' stdout > got
+cat exp
+cat got
+diff exp got
+count_test_results total=4 pass=2 fail=0 xpass=0 xfail=1 skip=1 error=0
+
+TEST_LOG_DRIVER_FLAGS="--no-comments" $MAKE -e check >stdout \
+ || { cat stdout; Exit 1; }
+cat stdout
+$EGREP -i "#.*all\\.test|a comment|(Tests|Shell) " stdout && Exit 1
+count_test_results total=4 pass=2 fail=0 xpass=0 xfail=1 skip=1 error=0
+
+# The "#"-prepended lines here shouldn't be parsed as test results.
+cat > all.test <<END
+1..1
+ok
+# ok
+#ok
+# not ok
+#not ok
+# Bail out!
+#Bail out!
+# SKIP
+#SKIP
+# TODO
+#TODO
+END
+
+$MAKE check >stdout || { cat stdout; Exit 1; }
+cat stdout
+count_test_results total=1 pass=1 fail=0 xpass=0 xfail=0 skip=0 error=0
+
+# Diagnostic without leading whitespace, or with extra leading whitespace,
+# is ok. Be laxer in the grepping checks, to allow for whitespace
+# normalization by the TAP driver.
+
+ws="[ $tab]"
+ws0p="${ws}*"
+ws1p="${ws}${ws0p}"
+
+cat > all.test <<END
+1..1
+ok 1
+#foo
+#bar${tab}
+# zardoz ${tab}
+# ${tab} ${tab}${tab}foo bar${tab}baz ${tab}
+END
+
+$MAKE check >stdout || { cat stdout; Exit 1; }
+cat stdout
+count_test_results total=1 pass=1 fail=0 xpass=0 xfail=0 skip=0 error=0
+
+grep "^# all.test:${ws0p}foo$" stdout
+grep "^# all.test:${ws0p}bar${ws0p}$" stdout
+grep "^# all.test:${ws1p}zardoz${ws0p}$" stdout
+grep "^# all.test:${ws1p}foo${ws1p}bar${ws1p}baz${ws0p}$" stdout
+
+test `grep -c '^# all\.test:' stdout` -eq 4
+
+:
diff --git a/tests/tap-empty-diagnostic.test b/tests/tap-empty-diagnostic.test
new file mode 100755
index 0000000..0edea13
--- /dev/null
+++ b/tests/tap-empty-diagnostic.test
@@ -0,0 +1,59 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support:
+# - empty diagnostic messages are discarder
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TEST_LOG_COMPILER = cat
+AM_TEST_LOG_DRIVER_FLAGS = --comments
+TESTS = all.test
+END
+
+sed 's/\$$//' > all.test <<END
+1..1$
+ok 1$
+#$
+# $
+#${tab}$
+# ${tab} $tab${tab}$
+END
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+$MAKE check >stdout || { cat stdout; Exit 1; }
+cat stdout
+
+grep '#.*all\.test' stdout && Exit 1
+grep "all\.test[ $tab]*:[ $tab]*$" stdout && Exit 1
+
+:
diff --git a/tests/tap-empty.test b/tests/tap-empty.test
new file mode 100755
index 0000000..565c134
--- /dev/null
+++ b/tests/tap-empty.test
@@ -0,0 +1,73 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support: the following situations should be flagged as errors:
+# - empty TAP input
+# - blank TAP input
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TEST_LOG_COMPILER = cat
+TESTS = all.test
+END
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+# Empty TAP input.
+: > empty.test
+
+# Blank TAP input (one empty line).
+echo > blank.test
+
+# Blank TAP input (one whitespace-only line).
+cat > white.test <<END
+ ${tab}
+END
+
+# Blank TAP input (few blank and whitespace-only lines).
+cat > white2.test <<END
+
+
+ ${tab} ${tab}${tab}
+
+${tab}
+
+END
+
+for input in empty blank white white2; do
+ cp $input.test all.test
+ $MAKE check >stdout && { cat stdout; Exit 1; }
+ cat stdout
+ count_test_results total=1 pass=0 fail=0 xpass=0 xfail=0 skip=0 error=1
+ grep '^ERROR: all\.test - missing test plan$' stdout
+done
+
+:
diff --git a/tests/tap-escape-directive.test b/tests/tap-escape-directive.test
new file mode 100755
index 0000000..fb333ba
--- /dev/null
+++ b/tests/tap-escape-directive.test
@@ -0,0 +1,57 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support:
+# - "escape" TODO and SKIP directives (by escaping the "#" character)
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TEST_LOG_COMPILER = cat
+TESTS = all.test
+END
+
+cat > all.test <<'END'
+1..2
+ok \# TODO
+ok \# SKIP
+END
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+$MAKE check >stdout || { cat stdout; Exit 1; }
+cat stdout
+
+count_test_results total=2 pass=2 fail=0 xpass=0 xfail=0 skip=0 error=0
+
+grep '^PASS: all\.test 1 .*TODO' stdout
+grep '^PASS: all\.test 2 .*SKIP' stdout
+
+:
diff --git a/tests/tap-exit.test b/tests/tap-exit.test
new file mode 100755
index 0000000..07209d1
--- /dev/null
+++ b/tests/tap-exit.test
@@ -0,0 +1,70 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support:
+# - an exit status != 0 of a test script causes an hard error
+# - the `--ignore-exit' option causes the TAP test driver to ignore
+# exit statuses of the test scripts.
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TESTS =
+END
+
+for st in 1 2 77 99; do
+ unindent > exit${st}.test <<END
+ #!/bin/sh
+ echo 1..1
+ echo ok 1
+ exit $st
+END
+ echo TESTS += exit${st}.test >> Makefile.am
+done
+
+chmod a+x *.test
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+$MAKE check >stdout && { cat stdout; Exit 1; }
+cat stdout
+count_test_results total=8 pass=4 fail=0 xpass=0 xfail=0 skip=0 error=4
+
+grep '^ERROR: exit1\.test - exited with status 1$' stdout
+grep '^ERROR: exit2\.test - exited with status 2$' stdout
+grep '^ERROR: exit77\.test - exited with status 77$' stdout
+grep '^ERROR: exit99\.test - exited with status 99$' stdout
+
+echo TEST_LOG_DRIVER_FLAGS = --ignore-exit >> Makefile
+$MAKE check >stdout || { cat stdout; Exit 1; }
+cat stdout
+count_test_results total=4 pass=4 fail=0 xpass=0 xfail=0 skip=0 error=0
+
+:
diff --git a/tests/tap-fancy.test b/tests/tap-fancy.test
new file mode 100755
index 0000000..68612de
--- /dev/null
+++ b/tests/tap-fancy.test
@@ -0,0 +1,151 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support: some unusual forms for valid TAP input.
+# See also related test 'tap-fancy2.test'.
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TEST_LOG_COMPILER = cat
+TESTS = all.test
+END
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+#
+# From manpage Test::Harness::TAP(3):
+#
+# ``Lines written to standard output matching /^(not )?ok\b/ must be
+# interpreted as test lines. All other lines must not be considered
+# test output.''
+#
+# Unfortunately, the exact format of TODO and SKIP directives is not as
+# clearly described in that manpage; but a simple reverse-engineering of
+# the prove(1) utility shows that it is probably given by the perl regex
+# /#\s*(TODO|SKIP)\b/.
+#
+
+cat > all.test <<END
+1..21
+
+ok? a question
+not ok? a question
+
+ok+plus
+not ok+plus
+
+ok-minus
+not ok-minus
+
+ok#55
+not ok#55
+
+ok${tab} ${tab}9
+ok ${tab}${tab} 10
+
+not ok${tab} ${tab}11
+not ok ${tab}${tab} 12
+
+ok# SKIP
+ok${tab}#SKIP--who cares?
+ok?#SKIP!
+ok!#SKIP?
+
+not ok# TODO
+not ok${tab}#TODO--who cares?
+not ok?#TODO!
+not ok!#TODO?
+
+ok~#TODO
+END
+
+$MAKE check >stdout && { cat stdout; Exit 1; }
+cat stdout
+
+count_test_results total=21 pass=6 fail=6 xfail=4 xpass=1 skip=4 error=0
+
+#
+# "Weird" characters support.
+#
+
+# The "#" character might cause confusion w.r.t. TAP directives (TODO,
+# SKIP), so we don't attempt to use it.
+weirdchars=\''"$!&()[]<>;^?*/@%=,.:'
+
+cat > all.test <<END
+1..6
+ok $weirdchars
+not ok $weirdchars
+ok $weirdchars # TODO
+not ok $weirdchars # TODO
+ok $weirdchars # SKIP
+Bail out! $weirdchars
+END
+
+$MAKE check >stdout && { cat stdout; Exit 1; }
+cat stdout
+
+count_test_results total=6 pass=1 fail=1 xfail=1 xpass=1 skip=1 error=1
+$FGREP "PASS: all.test 1 $weirdchars" stdout
+$FGREP "FAIL: all.test 2 $weirdchars" stdout
+$FGREP "XPASS: all.test 3 $weirdchars" stdout
+$FGREP "XFAIL: all.test 4 $weirdchars" stdout
+$FGREP "SKIP: all.test 5 $weirdchars" stdout
+$FGREP "ERROR: all.test - Bail out! $weirdchars" stdout
+
+#
+# Trailing backslashes does not confuse the parser.
+#
+
+bs='\'
+
+cat > all.test <<END
+1..6
+ok $bs
+not ok $bs
+ok # TODO $bs
+not ok # TODO $bs
+ok # SKIP $bs
+Bail out! $bs
+END
+
+$MAKE check >stdout && { cat stdout; Exit 1; }
+cat stdout
+
+count_test_results total=6 pass=1 fail=1 xfail=1 xpass=1 skip=1 error=1
+grep '^PASS: all\.test 1 \\$' stdout
+grep '^FAIL: all\.test 2 \\$' stdout
+grep '^XPASS: all\.test 3 # TODO \\$' stdout
+grep '^XFAIL: all\.test 4 # TODO \\$' stdout
+grep '^SKIP: all\.test 5 # SKIP \\$' stdout
+grep '^ERROR: all\.test - Bail out! \\$' stdout
+
+:
diff --git a/tests/tap-fancy2.test b/tests/tap-fancy2.test
new file mode 100755
index 0000000..b97d787
--- /dev/null
+++ b/tests/tap-fancy2.test
@@ -0,0 +1,147 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support: more unusual forms for valid TAP input.
+# See also related test 'tap-fancy.test'.
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TEST_LOG_COMPILER = cat
+TESTS = all.test
+END
+
+#
+# From manpage Test::Harness::TAP(3):
+#
+# ``Lines written to standard output matching /^(not )?ok\b/ must be
+# interpreted as test lines. All other lines must not be considered
+# test output.''
+#
+# Unfortunately, the exact format of TODO and SKIP directives is not as
+# clearly described in that manpage; but a simple reverse-engineering of
+# the prove(1) utility shows that it is probably given by the perl regex
+# /#\s*(TODO|SKIP)\b/.
+#
+
+# There are 34 values for $str ...
+for str in \
+ \' \
+ '"' \
+ '`' \
+ '#' \
+ '$' \
+ '!' \
+ '\' \
+ '/' \
+ '&' \
+ '%' \
+ '(' \
+ ')' \
+ '|' \
+ '^' \
+ '~' \
+ '?' \
+ '*' \
+ '+' \
+ '-' \
+ ',' \
+ ':' \
+ ';' \
+ '=' \
+ '<' \
+ '>' \
+ '@' \
+ '[' \
+ ']' \
+ '{' \
+ '}' \
+ '\\' \
+ '...' \
+ '?[a-zA-Z0-9]*' \
+ '*.*' \
+; do
+ # ... each of them add 1 pass, 1 fail, ...
+ echo "ok${str}"
+ echo "not ok${str}"
+ # ... and (generally) 4 skips, 4 xfails, and 4 xpasses ...
+ for settings in \
+ 'result="ok" directive=SKIP' \
+ 'result="not ok" directive=TODO' \
+ 'result="ok" directive=TODO' \
+ ; do
+ eval "$settings"
+ echo "${result}# ${directive}${str}"
+ # ... but 6 skips, 6 xpasses and 6 xfails are to be removed, since
+ # they might not work with $str = '#' or $str = '\' ...
+ if test x"$str" != x'#' && test x"$str" != x'\'; then
+ echo "${result}${str}#${directive}"
+ echo "${result}${str}# ${tab}${tab} ${directive}"
+ echo "${result}${str}#${directive}${str}"
+ fi
+ done
+done > all.test
+
+# ... so that we finally have:
+pass=34
+fail=34
+xfail=130 # = 4 * 34 - 6
+xpass=130 # = 4 * 34 - 6
+skip=130 # = 4 * 34 - 6
+error=0
+total=`expr $pass + $fail + $xfail + $xpass + $skip`
+
+# Even nastier! But accordingly to the specifics, it should still work.
+for result in 'ok' 'not ok'; do
+ echo "${result}{[(<#${tab}TODO>)]}" >> all.test
+done
+echo "ok{[(<#${tab}SKIP>)]}" >> all.test
+
+# We have to update some test counts.
+xfail=`expr $xfail + 1`
+xpass=`expr $xpass + 1`
+skip=`expr $skip + 1`
+total=`expr $total + 3`
+
+# And add the test plan!
+echo 1..$total >> all.test
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+$MAKE check >stdout && { cat stdout; Exit 1; }
+cat stdout
+
+$EGREP '^(PASS|FAIL|SKIP).*#.*TODO' stdout && Exit 1
+$EGREP '^X?(PASS|FAIL).*#.*SKIP' stdout && Exit 1
+
+count_test_results total=$total pass=$pass fail=$fail skip=$skip \
+ xpass=$xpass xfail=$xfail error=$error
+
+:
diff --git a/tests/tap-global-result.test b/tests/tap-global-result.test
new file mode 100755
index 0000000..d2e593e
--- /dev/null
+++ b/tests/tap-global-result.test
@@ -0,0 +1,128 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support:
+# - which global test result derives from different test results
+# mixed in a single script?
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TEST_LOG_COMPILER = cat
+TESTS = \
+ ok.test \
+ skip.test skipall.test \
+ fail.test fail2.test \
+ xpass.test xpass2.test \
+ error.test error2.test \
+ hodgepodge.test hodgepodge-all.test
+END
+
+cat > ok.test <<END
+1..3
+ok 1
+not ok 2 # TODO
+ok 3 # SKIP
+END
+
+cat > skip.test <<'END'
+1..3
+ok 1 # SKIP
+ok 2 # SKIP
+ok 3 # SKIP
+END
+
+cat > skipall.test <<'END'
+1..0 # SKIP
+not ok 1
+END
+
+cat > fail.test <<'END'
+1..1
+not ok 1
+END
+
+(sed '1s/.*/1..4/' ok.test && echo 'not ok 4') > fail2.test
+
+cat > xpass.test <<'END'
+1..1
+ok 1 # TODO
+END
+
+(sed '1s/.*/1..4/' ok.test && echo 'ok 4 # TODO') > xpass2.test
+
+echo 'Bail out!' > error.test
+(cat ok.test && echo 'Bail out!') > error2.test
+
+cat > hodgepodge.test <<'END'
+1..2
+not ok 1
+ok 2 # TODO
+Bail out!
+END
+
+cat > hodgepodge-all.test <<'END'
+1..4
+ok 1
+ok 2 # SKIP
+not ok 2 # TODO
+not ok 3
+ok 4 # TODO
+Bail out!
+END
+
+# TODO: add scripts with TAP errors (multiple plans, out-of-order
+# tests, etc).
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+$MAKE check >stdout && { cat stdout; Exit 1; }
+cat stdout
+
+for tst in ok skip skipall fail fail2 xpass xpass2 error error2 \
+ hodgepodge hodgepodge-all; do
+ sed -e 2q $tst.log > $tst.res
+done
+
+cat *.res # For debugging.
+
+grep '^PASS:' ok.res
+grep '^SKIP:' skip.res
+grep '^SKIP:' skipall.res
+grep '^FAIL:' fail.res
+grep '^FAIL:' fail2.res
+grep '^FAIL:' xpass.res
+grep '^FAIL:' xpass2.res
+grep '^ERROR:' error.res
+grep '^ERROR:' error2.res
+grep '^ERROR:' hodgepodge.res
+grep '^ERROR:' hodgepodge-all.res
+
+:
diff --git a/tests/tap-html.test b/tests/tap-html.test
new file mode 100755
index 0000000..12dcb36
--- /dev/null
+++ b/tests/tap-html.test
@@ -0,0 +1,93 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support:
+# - "check-html" and "recheck-html" targets
+# - reStructuredText -> HTML conversion
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TEST_LOG_COMPILER = cat
+TESTS = ok.test ko.test
+END
+
+cat > ok.test << 'END'
+1..3
+ok 1 - expected success
+not ok 2 - expectred failure # TODO
+ok 3 - skipped test # SKIP
+END
+
+cat > ko.test << 'END'
+1..2
+not ok 1 - unexpected failure
+ok 2 - spurious success # TODO
+Bail out! hard error
+END
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+no_rst2html_error ()
+{
+ $EGREP '(test-suite\.log:[0-9]|WARNING|SEVERE)' output && Exit 1
+ :
+}
+
+# Overriding TESTS should work with check-html.
+TESTS=ok.test $MAKE -e check-html >output 2>&1 || { cat output; Exit 1; }
+cat output
+no_rst2html_error
+test -f test-suite.html
+
+mv output stdout
+count_test_results total=3 pass=1 fail=0 xpass=0 xfail=1 skip=1 error=0
+
+: > older
+$sleep
+$MAKE check-html >output 2>&1 && { cat output; Exit 1; }
+cat output
+no_rst2html_error
+# test-suite.html should be remake if if it was up-to-date.
+test -f test-suite.html
+is_newest test-suite.html older
+
+mv output stdout
+count_test_results total=6 pass=1 fail=1 xpass=1 xfail=1 skip=1 error=1
+
+rm -f ok.test
+$MAKE recheck-html >output 2>&1 && { cat output; Exit 1; }
+cat output
+no_rst2html_error
+
+mv output stdout
+count_test_results total=3 pass=0 fail=1 xpass=1 xfail=0 skip=0 error=1
+
+:
diff --git a/tests/tap-log.test b/tests/tap-log.test
new file mode 100755
index 0000000..37b5956
--- /dev/null
+++ b/tests/tap-log.test
@@ -0,0 +1,174 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support:
+# - log file creation
+# - log file removal
+# - stdout and stderr of a TAP-generating test script go in its log file
+# - TEST_SUITE_LOG redefinition, at either automake or make time
+# - VERBOSE environment variable support
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TESTS = pass.test skip.test xfail.test fail.test xpass.test error.test
+TEST_SUITE_LOG = global.log
+END
+
+# Custom markers, for use in grepping checks.
+cmarker=::: # comment marker
+pmarker=%%% # plain maker
+
+cat > pass.test <<END
+#! /bin/sh -e
+echo 1..1
+echo "$pmarker pass $pmarker" >&2
+echo "# $cmarker pass $cmarker" >&2
+echo "ok 1"
+END
+
+cat > skip.test <<END
+#! /bin/sh -e
+echo 1..1
+echo "$pmarker skip $pmarker"
+echo "# $cmarker skip $cmarker"
+echo "ok 1 # SKIP"
+END
+
+cat > xfail.test <<END
+#! /bin/sh -e
+echo 1..1
+echo "$pmarker xfail $pmarker" >&2
+echo "# $cmarker xfail $cmarker" >&2
+echo "not ok 1 # TODO"
+END
+
+cat > fail.test <<END
+#! /bin/sh -e
+echo 1..1
+echo "$pmarker fail $pmarker"
+echo "# $cmarker fail $cmarker"
+echo "not ok 1"
+END
+
+cat > xpass.test <<END
+#! /bin/sh -e
+echo 1..1
+echo "$pmarker xpass $pmarker" >&2
+echo "# $cmarker xpass $cmarker" >&2
+echo "ok 1 # TODO"
+END
+
+cat > error.test <<END
+#! /bin/sh -e
+echo 1..1
+echo "$pmarker error $pmarker"
+echo "# $cmarker error $cmarker"
+echo 'Bail out!'
+END
+
+chmod a+x *.test
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+TEST_SUITE_LOG=my.log $MAKE -e check && Exit 1
+ls -l # For debugging.
+test ! -f test-suite.log
+test ! -f global.log
+test -f my.log
+st=0
+for result in pass fail xfail xpass skip error; do
+ cat $result.log # For debugging.
+ $FGREP "$pmarker $result $pmarker" $result.log || st=1
+ $FGREP "$cmarker $result $cmarker" $result.log || st=1
+done
+test $st -eq 0 || Exit 1
+cat my.log # For debugging.
+for result in fail xpass skip error; do
+ cat $result.log # For debugging.
+ $FGREP "$pmarker $result $pmarker" my.log || st=1
+ $FGREP "$cmarker $result $cmarker" my.log || st=1
+done
+test `$FGREP -c "$pmarker" my.log` -eq 4
+test `$FGREP -c "$cmarker" my.log` -eq 4
+
+# Successful test scripts shouldn't be mentioned in the global log.
+$EGREP '(^pass|[^x]pass|xfail)\.test' my.log && Exit 1
+# But failing and skipped one should.
+$FGREP 'skip.test' my.log
+$FGREP 'fail.test' my.log
+$FGREP 'xpass.test' my.log
+$FGREP 'error.test' my.log
+
+touch error2.log test-suite.log global.log
+TEST_SUITE_LOG=my.log $MAKE -e mostlyclean
+ls -l # For debugging.
+test ! -f my.log
+test ! -f pass.log
+test ! -f fail.log
+test ! -f xfail.log
+test ! -f xpass.log
+test ! -f skip.log
+test ! -f error.log
+test -f error2.log
+test -f test-suite.log
+test -f global.log
+
+rm -f *.log
+
+VERBOSE=yes $MAKE check >stdout && { cat stdout; Exit 1; }
+cat stdout
+cat global.log
+test ! -f my.log
+test ! -f test-suite.log
+# Check that VERBOSE causes the global testsuite log to be
+# emitted on stdout.
+out=`cat stdout`
+log=`cat global.log`
+case $out in *"$log"*) ;; *) Exit 1;; esac
+
+touch error2.log test-suite.log my.log
+$MAKE mostlyclean
+ls -l # For debugging.
+test ! -f global.log
+test ! -f pass.log
+test ! -f fail.log
+test ! -f xfail.log
+test ! -f xpass.log
+test ! -f skip.log
+test ! -f error.log
+test -f error2.log
+test -f test-suite.log
+test -f my.log
+
+rm -f *.log
+
+
+:
diff --git a/tests/tap-merge-stdout-stderr.test
b/tests/tap-merge-stdout-stderr.test
new file mode 100755
index 0000000..e73f0ec
--- /dev/null
+++ b/tests/tap-merge-stdout-stderr.test
@@ -0,0 +1,85 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support:
+# - The Automake TAP driver has an option that instruct it to read TAP
+# input also from the stderr of the test command, not only its stdout.
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+AM_TEST_LOG_DRIVER_FLAGS = --comments --merge
+TESTS = all.test
+END
+
+: > all.test # Updated later.
+chmod a+x all.test
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+cat > all.test <<END
+#!/bin/sh
+echo 1..4
+echo ok 1 >&2
+echo ok 2
+echo "not ok 3 # TODO" >&2
+echo "ok 4 # SKIP"
+echo "# foo foo foo" >&2
+END
+
+$MAKE check >stdout || { cat stdout; Exit 1; }
+cat stdout
+
+count_test_results total=4 pass=2 fail=0 xpass=0 xfail=1 skip=1 error=0
+grep '^# all\.test: foo foo foo' stdout
+
+cat > all.test <<END
+#!/bin/sh
+echo 1..1
+echo ok 1
+echo 'Bail out!' >&2
+END
+
+$MAKE check >stdout && { cat stdout; Exit 1; }
+cat stdout
+
+count_test_results total=2 pass=1 fail=0 xpass=0 xfail=0 skip=0 error=1
+
+# See that the option `--no-merge' can override the effect of `--merge'.
+
+TEST_LOG_DRIVER_FLAGS=--no-merge $MAKE -e check >stdout \
+ || { cat stdout; Exit 1; }
+cat stdout
+
+count_test_results total=1 pass=1 fail=0 xpass=0 xfail=0 skip=0 error=0
+
+
+
+:
diff --git a/tests/tap-message-0.test b/tests/tap-message-0.test
new file mode 100755
index 0000000..9f16171
--- /dev/null
+++ b/tests/tap-message-0.test
@@ -0,0 +1,84 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support:
+# - having "0" as a test description (or TODO or SKIP message) should
+# be supported
+# Note that a bug in some versions of TAP::Parser causes this not to be
+# generally true!
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TEST_LOG_COMPILER = cat
+TESTS = all.test
+END
+
+cat > all.test << 'END'
+1..10
+ok 1 0
+ok - 0
+not ok 3 0 # TODO
+not ok - 0 # TODO
+ok 5 0 # SKIP
+ok - 0 # SKIP
+not ok 7 # TODO 0
+ok 8 # SKIP 0
+not ok 9 0 # TODO 0
+ok 10 0 # SKIP 0
+END
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+$MAKE check >stdout || { cat stdout; Exit 1; }
+cat stdout
+
+count_test_results total=10 pass=2 fail=0 xpass=0 xfail=4 skip=4 error=0
+
+cat > exp << 'END'
+PASS: all.test 1 0
+PASS: all.test 2 - 0
+XFAIL: all.test 3 0 # TODO
+XFAIL: all.test 4 - 0 # TODO
+SKIP: all.test 5 0 # SKIP
+SKIP: all.test 6 - 0 # SKIP
+XFAIL: all.test 7 # TODO 0
+SKIP: all.test 8 # SKIP 0
+XFAIL: all.test 9 0 # TODO 0
+SKIP: all.test 10 0 # SKIP 0
+END
+
+$FGREP ': all.test' stdout > got
+
+cat exp
+cat got
+diff exp got
+
+:
diff --git a/tests/tap-more.test b/tests/tap-more.test
new file mode 100755
index 0000000..72fcaac
--- /dev/null
+++ b/tests/tap-more.test
@@ -0,0 +1,160 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# More on TAP support:
+# - more than one TAP-generating test script in $(TESTS)
+# - VPATH builds
+# - use with parallel make (if supported)
+# - basic use of diagnostic messages (lines beginning with "#")
+# - flags for TAP driver defined through AC_SUBST in configure.ac
+# - messages generated by the testsuite harness reference the
+# correct test script(s)
+# - "make distcheck" works
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_SUBST([AM_TEST_LOG_DRIVER_FLAGS], ['--comments'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TESTS = 1.test 2.test 3.test
+EXTRA_DIST = $(TESTS) tap-driver
+END
+
+cat > 1.test <<'END'
+#! /bin/sh
+echo 1..2
+echo ok 1 - mu
+if test -f not-skip; then
+ echo "not ok 2 zardoz"
+else
+ echo "ok 2 zardoz # SKIP"
+fi
+END
+
+cat > 2.test <<'END'
+#! /bin/sh
+echo 1..3
+echo "ok"
+echo "not ok # TODO not implemented"
+echo "ok 3"
+END
+
+cat > 3.test <<END
+#! /bin/sh
+echo 1..1
+echo ok - blah blah blah
+echo '# Some diagnostic'
+if test -f bail-out; then
+ echo 'Bail out! Kernel Panic'
+else
+ :
+fi
+END
+
+chmod a+x [123].test
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+# Try a VPATH and by default serial build first, and then an in-tree
+# and by default parallel build.
+for try in 0 1; do
+
+ if test $try -eq 0; then
+ # VPATH serial build.
+ mkdir build
+ cd build
+ srcdir=..
+ run_make=$MAKE
+ elif test $try -eq 1; then
+ # In-tree parallel build.
+ srcdir=.
+ case $MAKE in
+ *\ -j*)
+ # Degree of parallelism already specified by the user: do
+ # not override it.
+ run_make=$MAKE
+ ;;
+ *)
+ # Some make implementations (e.g., HP-UX) don't grok `-j',
+ # some require no space between `-j' and the number of jobs
+ # (e.g., older GNU make versions), and some *do* require a
+ # space between `-j' and the number of jobs (e.g., Solaris
+ # dmake). We need a runtime test to see what works.
+ echo 'all:' > Makefile
+ for run_make in "$MAKE -j3" "$MAKE -j 3" "$MAKE"; do
+ $run_make && break
+ done
+ rm -f Makefile
+ ;;
+ esac
+ else
+ fatal_ "internal error, invalid value of '$try' for \$try"
+ fi
+
+ $srcdir/configure
+ ls -l # For debugging.
+
+ # Success.
+ $run_make check >stdout || { cat stdout; Exit 1; }
+ cat stdout
+ count_test_results total=6 pass=4 fail=0 xpass=0 xfail=1 skip=1 error=0
+ grep '^PASS: 1\.test 1 - mu$' stdout
+ grep '^SKIP: 1\.test 2 zardoz # SKIP$' stdout
+ test `$FGREP -c '1.test' stdout` -eq 2
+ grep '^PASS: 2\.test 1$' stdout
+ grep '^XFAIL: 2\.test 2 # TODO not implemented$' stdout
+ grep '^PASS: 2\.test 3$' stdout
+ test `$FGREP -c '2.test' stdout` -eq 3
+ grep '^PASS: 3\.test 1 - blah blah blah$' stdout
+ grep '^# 3\.test: Some diagnostic$' stdout
+ test `$FGREP -c '3.test' stdout` -eq 2
+
+ # Failure.
+ : > not-skip
+ : > bail-out
+ $run_make check >stdout && { cat stdout; Exit 1; }
+ cat stdout
+ count_test_results total=7 pass=4 fail=1 xpass=0 xfail=1 skip=0 error=1
+ grep '^PASS: 1\.test 1 - mu$' stdout
+ grep '^FAIL: 1\.test 2 zardoz$' stdout
+ test `$FGREP -c '1.test' stdout` -eq 2
+ grep '^PASS: 2\.test 1$' stdout
+ grep '^XFAIL: 2\.test 2 # TODO not implemented$' stdout
+ grep '^PASS: 2\.test 3$' stdout
+ test `$FGREP -c '2.test' stdout` -eq 3
+ grep '^PASS: 3\.test 1 - blah blah blah$' stdout
+ grep '^# 3\.test: Some diagnostic$' stdout
+ grep '^ERROR: 3\.test - Bail out! Kernel Panic$' stdout
+ test `$FGREP -c '3.test' stdout` -eq 3
+
+ cd $srcdir
+
+done
+
+$MAKE distcheck
+
+:
diff --git a/tests/tap-more2.test b/tests/tap-more2.test
new file mode 100755
index 0000000..810f9b8
--- /dev/null
+++ b/tests/tap-more2.test
@@ -0,0 +1,93 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# More on TAP support:
+# - more LOG_COMPILER at once for TAP tests
+# - binary programs in $(TESTS)
+# - interactions with `check_*' variables
+
+required='cc native'
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_PROG_CC
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_EXTENSIONS = .sh .tap
+
+tap_driver = $(PERL) $(srcdir)/tap-driver
+
+LOG_DRIVER = $(tap_driver)
+SH_LOG_DRIVER = $(tap_driver)
+TAP_LOG_DRIVER = $(tap_driver)
+
+TAP_LOG_COMPILER = cat
+SH_LOG_COMPILER = $(SHELL)
+LOG_COMPILER =
+
+EXTRA_DIST = baz.tap
+
+check_SCRIPTS = bar.sh
+bar.sh: Makefile
+ echo '#!/bin/sh' > address@hidden
+ echo 'echo 1..1' >> address@hidden
+ echo 'echo "not ok 1 # TODO"' >> address@hidden
+ chmod a-w address@hidden && mv -f address@hidden $@
+CLEANFILES = bar.sh
+
+check_PROGRAMS = foo-test
+foo_test_SOURCES = foo.c
+
+TESTS = foo-test $(check_SCRIPTS) baz.tap
+
+EXTRA_DIST += tap-driver
+END
+
+cat > foo.c <<'END'
+#include <stdio.h>
+int main (void)
+{
+ printf ("1..1\n");
+ printf ("ok 1\n");
+ return 0;
+}
+END
+
+cat > baz.tap << 'END'
+1..1
+ok 1 # SKIP
+END
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+for target in check distcheck; do
+ $MAKE $target >stdout || { cat stdout; Exit 1; }
+ cat stdout
+ count_test_results total=3 pass=1 fail=0 xpass=0 xfail=1 skip=1 error=0
+done
+
+:
diff --git a/tests/tap-no-merge-stdout-stderr.test
b/tests/tap-no-merge-stdout-stderr.test
new file mode 100755
index 0000000..c2995b2
--- /dev/null
+++ b/tests/tap-no-merge-stdout-stderr.test
@@ -0,0 +1,66 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support:
+# - By default, TAP input is only from the stdout (and not the stderr)
+# of the test command.
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TEST_LOG_DRIVER_FLAGS = --comments
+TESTS = all.test
+END
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+cat > all.test <<END
+#!/bin/sh
+echo 1..2
+echo 'not ok' >&2
+echo 'not ok 1 # TODO' >&2
+echo 'ok 1'
+echo '# foo foo foo'
+echo '# bar bar bar' >&2
+echo 'ok 44 # TODO' >&2
+echo 'Bail out!' >&2
+echo 'ok 2 # SKIP'
+END
+
+chmod a+x all.test
+
+$MAKE check >stdout || { cat stdout; Exit 1; }
+cat stdout
+
+count_test_results total=2 pass=1 fail=0 xpass=0 xfail=0 skip=1 error=0
+$FGREP 'foo foo foo' stdout
+$FGREP 'bar bar bar' stdout && Exit 1
+
+:
diff --git a/tests/tap-no-spurious-summary.test
b/tests/tap-no-spurious-summary.test
new file mode 100755
index 0000000..4c358da
--- /dev/null
+++ b/tests/tap-no-spurious-summary.test
@@ -0,0 +1,65 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support:
+# - `:test-results:' directives in test scripts' output doesn't
+# originate spurious results in the testsuite summary
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TEST_LOG_COMPILER = cat
+TESTS = all.test
+END
+
+cat > all.test <<'END'
+1..1
+:test-result: PASS
+:test-result: FAIL
+:test-result: XPASS
+:test-result: XFAIL
+:test-result: SKIP
+:test-result: ERROR
+:test-result: UNKNOWN
+ok 1
+END
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+$MAKE check >stdout || { cat stdout; Exit 1; }
+cat stdout
+
+count_test_results total=1 pass=1 fail=0 xpass=0 xfail=0 skip=0 error=0
+
+for result in PASS FAIL XPASS XFAIL SKIP ERROR UNKNOWN; do
+ grep "^ *:test-result: $result$" all.log
+done
+
+:
diff --git a/tests/tap-no-spurious.test b/tests/tap-no-spurious.test
new file mode 100755
index 0000000..cfe7dbd
--- /dev/null
+++ b/tests/tap-no-spurious.test
@@ -0,0 +1,114 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support:
+# - don't spuriously recognize lines that are "almost" TAP lines as
+# real TAP lines
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TEST_LOG_COMPILER = cat
+TESTS = all.test
+END
+
+echo 1..5 > all.test
+
+# The only recognized directives are "TODO" and "SKIP".
+# So the following should count as passed tests.
+cat >> all.test <<END
+ok 1 # XFAIL
+ok 2 # SKIPPED
+ok 3 # TO DO
+ok 4 # TODOALL
+ok 5 # FIXME
+END
+
+# According to documentation of Test::Harness::TAP(3):
+#
+# ``Lines written to standard output matching /^(not )?ok\b/
+# must be interpreted as test lines. All other lines must
+# not be considered test output.''
+
+cat >> all.test <<END
+ ok
+ ok 1
+${tab}ok
+${tab}ok 1
+ok1
+ok23
+ not ok
+ not ok 1
+${tab}not ok
+${tab}not ok 1
+not ok1
+not ok23
+notok
+notok 1
+not${tab}ok
+not${tab}ok 1
+not ok
+not ok 1
+no ok
+no ok 1
+# ok
+# not ok
+# ok 1
+# not ok 1
+#ok
+#not ok
+#ok 1
+#not ok 1
+END
+
+# The prove(1) utility doesn't bail out on these, so our driver
+# shouldn't either.
+cat >> all.test <<'END'
+bailout
+bailout!
+bail out
+bail out!
+Bailout
+Bailout!
+Bail out
+ Bail out
+ Bail out!
+#Bail out!
+# Bail out!
+END
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+$MAKE check >stdout || { cat stdout; Exit 1; }
+cat stdout
+
+count_test_results total=5 pass=5 fail=0 xpass=0 xfail=0 skip=0 error=0
+
+:
diff --git a/tests/tap-not-ok-skip.test b/tests/tap-not-ok-skip.test
new file mode 100755
index 0000000..57b8d7a
--- /dev/null
+++ b/tests/tap-not-ok-skip.test
@@ -0,0 +1,57 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support:
+# - a "not ok # SKIP" line should count as a failure, for consistency
+# with the prove(1) utility.
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TEST_LOG_COMPILER = cat
+TESTS = all.test
+END
+
+cat > all.test <<'END'
+1..4
+not ok # SKIP
+not ok 2 # SKIP
+not ok - foo # SKIP
+not ok 4 - bar # SKIP
+END
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+$MAKE check >stdout && { cat stdout; Exit 1; }
+cat stdout
+
+count_test_results total=4 pass=0 fail=4 skip=0 xpass=0 xfail=0 error=0
+
+:
diff --git a/tests/tap-numeric-description.test
b/tests/tap-numeric-description.test
new file mode 100755
index 0000000..67b3733
--- /dev/null
+++ b/tests/tap-numeric-description.test
@@ -0,0 +1,86 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP result lines whose description is a number.
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TEST_LOG_COMPILER = cat
+TESTS = all.test
+END
+
+# Some random numbers to be used as test names.
+# These are selected so that $n != n for every n >= 1.
+set 5 7 45 11 7 3 6 9 1000 1
+# $1 $2 $3 $4 $5 $6 $7 $8 $9 $10
+
+cat > all.test <<END
+1..10
+ok 1 ${1}
+ok - ${2}
+not ok 3 ${3}
+not ok - ${4}
+ok 5 ${5} # SKIP
+ok - ${6} # SKIP
+not ok 7 ${7} # TODO
+not ok - ${8} # TODO
+ok 9 ${9} # TODO
+ok - ${10} # TODO
+END
+
+cat > exp <<END
+PASS: all.test 1 ${1}
+PASS: all.test 2 - ${2}
+FAIL: all.test 3 ${3}
+FAIL: all.test 4 - ${4}
+SKIP: all.test 5 ${5} # SKIP
+SKIP: all.test 6 - ${6} # SKIP
+XFAIL: all.test 7 ${7} # TODO
+XFAIL: all.test 8 - ${8} # TODO
+XPASS: all.test 9 ${9} # TODO
+XPASS: all.test 10 - ${10} # TODO
+END
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+$MAKE check >stdout && { cat stdout; Exit 1; }
+cat stdout
+
+count_test_results total=10 pass=2 fail=2 xpass=2 xfail=2 skip=2 error=0
+
+$FGREP ': all.test' stdout > got
+
+cat exp
+cat got
+diff exp got
+
+
+:
diff --git a/tests/tap-out-of-order.test b/tests/tap-out-of-order.test
new file mode 100755
index 0000000..2639e2c
--- /dev/null
+++ b/tests/tap-out-of-order.test
@@ -0,0 +1,98 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support:
+# - out-of-order test results
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TEST_LOG_COMPILER = cat
+TESTS = a.test b.test c.test d.test
+END
+
+cat > a.test <<END
+1..3
+ok
+not ok 2 # TODO
+ok 71
+END
+
+cat > b.test <<END
+1..5
+ok 1
+ok 2
+ok 4 foo # SKIP mu
+not ok 2 bar # TODO um
+ok 5
+END
+
+cat > c.test <<END
+1..4
+ok 1
+ok foo
+ok 4 - zardoz
+not ok 3
+END
+
+# This is documented as an "test out of sequence" error in the
+# TAP::Parser documentation. Keep it in sync with the example
+# there.
+cat > d.test <<'END'
+1..5
+ok 1
+ok 2
+ok 15
+ok 16
+ok 17
+END
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+$MAKE check >stdout && { cat stdout; Exit 1; }
+cat stdout
+
+count_test_results total=17 pass=8 fail=0 xpass=0 xfail=1 skip=0 error=8
+
+test `$FGREP -c ': a.test' stdout` -eq 3
+test `$FGREP -c ': b.test' stdout` -eq 5
+test `$FGREP -c ': c.test' stdout` -eq 4
+test `$FGREP -c ': d.test' stdout` -eq 5
+
+grep '^ERROR: a\.test 71 # OUT-OF-ORDER (expecting 3)$' stdout
+grep '^ERROR: b\.test 4 foo # OUT-OF-ORDER (expecting 3)$' stdout
+grep '^ERROR: b\.test 2 bar # OUT-OF-ORDER (expecting 4)$' stdout
+grep '^ERROR: c\.test 4 - zardoz # OUT-OF-ORDER (expecting 3)$' stdout
+grep '^ERROR: c\.test 3 # OUT-OF-ORDER (expecting 4)$' stdout
+grep '^ERROR: d\.test 15 # OUT-OF-ORDER (expecting 3)$' stdout
+grep '^ERROR: d\.test 16 # OUT-OF-ORDER (expecting 4)$' stdout
+grep '^ERROR: d\.test 17 # OUT-OF-ORDER (expecting 5)$' stdout
+
+:
diff --git a/tests/tap-passthrough-exit.test b/tests/tap-passthrough-exit.test
new file mode 100755
index 0000000..98213a3
--- /dev/null
+++ b/tests/tap-passthrough-exit.test
@@ -0,0 +1,79 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support:
+# - non-success exit status of a test script is reported in the
+# log file
+# See also related test 'tap-passthrough.test'.
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TEST_LOG_COMPILER = $(SHELL)
+TESTS =
+END
+
+exit_statuses='1 2 77 99'
+
+for e in $exit_statuses; do
+ unindent > exit-$e.test <<END
+ #!/bin/sh
+ echo 1..1
+ echo ok 1
+ exit $e
+END
+ echo TESTS += exit-$e.test >> Makefile.am
+done
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+st=0
+$MAKE check || st=$?
+for e in $exit_statuses; do cat exit-$e.log; done
+cat test-suite.log
+test $st -gt 0 || Exit 1
+
+for e in $exit_statuses; do
+ for log in exit-$e.log test-suite.log; do
+ grep "^ERROR: exit-$e\\.test - exited with status $e$" $log
+ done
+done
+
+st=0
+env TEST_LOG_DRIVER_FLAGS='--ignore-exit' $MAKE -e check || st=$?
+for e in $exit_statuses; do cat exit-$e.log; done
+cat test-suite.log
+test $st -eq 0 || Exit 1
+
+for e in $exit_statuses; do
+ grep "exit-$e\\.test - exited with status $e$" exit-$e.log
+done
+
+:
diff --git a/tests/tap-passthrough.test b/tests/tap-passthrough.test
new file mode 100755
index 0000000..ab9b3e5
--- /dev/null
+++ b/tests/tap-passthrough.test
@@ -0,0 +1,202 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support:
+# - all input (valid TAP lines, invalid TAP lines, non-TAP lines)
+# are passed through in the log file
+# - TAP errors are reported in the log file too
+# See also related test 'tap-passthrough-exit.test'.
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TEST_LOG_COMPILER = cat
+TESTS =
+END
+
+weirdchars=\''"address@hidden&()[]<>#;,:.^?*/'
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+#
+# Only successful tests.
+#
+
+# The whitespace in this test might be normalized in the testsuite
+# progress output, but should be copied verbatim in the log files.
+cat > ok.test <<END
+1..6
+TAP plan in the previous line.
+ok${tab}
+ok 2
+ok - foo
+ok 4 - x
+ This is not a TAP line, but should still be copied in the log file!
+# some diagnostic${tab}
+not ok # TODO low priority
+ok # SKIP who cares?
+This is not in test-suite.log, since ok.test does not fail.
+$weirdchars
+END
+
+st=0
+TESTS=ok.test $MAKE -e check || st=$?
+cat ok.log
+cat test-suite.log
+test $st -eq 0 || Exit 1
+
+for rx in \
+ '1\.\.6' \
+ 'TAP plan in the previous line\.' \
+ "ok${tab}" \
+ 'ok 2' \
+ 'ok - foo' \
+ 'ok 4 - x' \
+ ' This is not a TAP line, but should still be copied in the log file!' \
+ "# some diagnostic${tab}" \
+ 'not ok # TODO low priority' \
+ 'ok # SKIP who cares?' \
+ 'This is not in test-suite\.log, since ok\.test does not fail\.' \
+; do
+ grep "^$rx$" ok.log
+done
+$FGREP "$weirdchars" ok.log
+
+$FGREP 'ok.test' test-suite.log && Exit 1
+$FGREP 'not in test-suite.log' test-suite.log && Exit 1
+
+#
+# Mixed failing/successful tests.
+#
+
+cat > tiny.test <<END
+1..1
+ok
+END
+
+cat > ok.test <<END
+1..1
+ok
+local only
+END
+
+cat > ko.test <<END
+1..5
+foo foo foo
+ok${tab}
+ok 2
+not ok - foo
+not ok 4 - x
+# diagnostic ko
+ bar${tab}bar${tab}bar
+ok # TODO dunno
+$weirdchars
+END
+
+cat > bail.test <<END
+Bail out! Test is taking too long!
+END
+
+cat > skip.test <<END
+1..0 # Skipped: WWW::Mechanize not installed
+END
+
+cat > err.test <<END
+1..3
+ok 1
+Invalid test count
+ok 23
+Misplaced plan
+1..13
+ok
+Extra test
+ok
+Last line
+END
+
+st=0
+env TESTS='tiny.test ok.test ko.test bail.test skip.test err.test' \
+ $MAKE -e check || st=$?
+cat tiny.log
+cat ok.log
+cat ko.log
+cat bail.log
+cat skip.log
+cat err.log
+cat test-suite.log
+test $st -gt 0 || Exit 1
+
+grep '^1\.\.1$' tiny.log
+grep '^ok$' tiny.log
+
+grep '^local only$' ok.log
+grep 'local only' test-suite.log && Exit 1
+
+for rx in \
+ '1\.\.5' \
+ 'foo foo foo' \
+ "ok${tab}" \
+ 'ok 2' \
+ 'not ok - foo' \
+ 'not ok 4 - x' \
+ '# diagnostic ko' \
+ " bar${tab}bar${tab}bar" \
+ 'ok # TODO dunno' \
+; do
+ grep "^$rx$" ko.log
+ grep "^$rx$" test-suite.log
+done
+$FGREP "$weirdchars" ko.log
+$FGREP "$weirdchars" test-suite.log
+
+for log in bail.log test-suite.log; do
+ grep '^Bail out! Test is taking too long!$' $log
+done
+for log in skip.log test-suite.log; do
+ grep '^1\.\.0 # Skipped: WWW::Mechanize not installed$' $log
+done
+
+for rx in \
+ '^1\.\.3$' \
+ '^Invalid test count$' \
+ '^ok 23$' \
+ '^Misplaced plan$' \
+ '^1\.\.13$' \
+ '^ERROR:.* test plan in middle of output' \
+ '^Extra test$' \
+ '^Last line$' \
+ '^ERROR:.* [tT]oo many tests run.*expected 3, got 4' \
+ '^ERROR:.* err\.test 23 .*OUT[ -]OF[ -]ORDER.*expecting 2' \
+; do
+ grep "$rx" err.log
+ grep "$rx" test-suite.log
+done
+
+:
diff --git a/tests/tap-plan-corner.test b/tests/tap-plan-corner.test
new file mode 100755
index 0000000..7f46263
--- /dev/null
+++ b/tests/tap-plan-corner.test
@@ -0,0 +1,81 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support:
+# - some corner cases for TAP plan
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TEST_LOG_COMPILER = cat
+TESTS = all.test
+END
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+cat > leading-repeated-1.test <<END
+1..1
+1..1
+ok 1
+END
+
+cat > trailing-repeated-1.test <<END
+ok 1
+1..1
+1..1
+END
+
+cat > leading-repeated-2.test <<END
+1..2
+ok 1
+1..2
+ok 2
+END
+
+cat > trailing-repeated-2.test <<END
+ok 1
+1..2
+ok 2
+1..2
+END
+
+for i in 1 2; do
+ for kind in leading trailing; do
+ cp $kind-repeated-$i.test all.test
+ $MAKE check >stdout && { cat stdout; Exit 1; }
+ cat stdout
+ count_test_results total=`expr $i + 2` pass=$i fail=0 \
+ xpass=0 xfail=0 skip=0 error=2
+ grep '^ERROR: all\.test - test plan in middle of output$' stdout
+ grep '^ERROR: all\.test - multiple test plans$' stdout
+ done
+done
+
+:
diff --git a/tests/tap-plan-corner2.test b/tests/tap-plan-corner2.test
new file mode 100755
index 0000000..155cf85
--- /dev/null
+++ b/tests/tap-plan-corner2.test
@@ -0,0 +1,67 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support:
+# - more corner cases for TAP plan
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TEST_LOG_COMPILER = cat
+TESTS = all.test
+END
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+# The leading blank line is meant.
+cat > leading-blank.test <<END
+
+1..2
+ok 1
+ok 2
+END
+
+# The trailing blank line is meant.
+cat > trailing-blank.test <<END
+ok 1
+ok 2
+1..2
+
+END
+
+for kind in leading trailing; do
+ cp $kind-blank.test all.test
+ $MAKE check >stdout && { cat stdout; Exit 1; }
+ cat stdout
+ count_test_results total=3 pass=2 fail=0 xpass=0 xfail=0 skip=0 error=1
+ grep '^ERROR: all\.test - test plan in middle of output$' stdout
+done
+
+:
diff --git a/tests/tap-plan-errors.test b/tests/tap-plan-errors.test
new file mode 100755
index 0000000..539cb8f
--- /dev/null
+++ b/tests/tap-plan-errors.test
@@ -0,0 +1,110 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support: the following situations should be flagged as errors:
+# - unmatched test plan (too few tests run)
+# - misplaced test plan
+# - multiple test plans
+# - missing test plan
+# Checks about unplanned tests are performed in 'tap-unplanned.test'.
+# More checks about corner-cases in TAP plans are performed in
+# 'tap-plan-corner.test' and 'tap-plan-corner2.test'.
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TEST_LOG_COMPILER = cat
+TESTS = foo.test
+END
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+my_check ()
+{
+ cat > foo.test
+ test -n "$err" || fatal_ "\$err not set before calling my_check"
+ cat foo.test # For debugging.
+ $MAKE check >stdout && { cat stdout; Exit 1; }
+ cat stdout
+ count_test_results "$@"
+ grep "^ERROR: foo\\.test - $err$" stdout
+ unset err
+}
+
+err='too few tests run (expected 2, got 1)'
+my_check total=2 pass=1 fail=0 xpass=0 xfail=0 skip=0 error=1 <<END
+1..2
+ok 1
+END
+
+err='too few tests run (expected 12, got 3)'
+my_check total=4 pass=2 fail=0 xpass=0 xfail=1 skip=0 error=1 <<END
+ok 1
+ok 2
+not ok 3 # TODO
+1..12
+END
+
+err='too few tests run (expected 1, got 0)'
+my_check total=1 pass=0 fail=0 xpass=0 xfail=0 skip=0 error=1 <<END
+1..1
+END
+
+err='test plan in middle of output'
+my_check total=3 pass=2 fail=0 xpass=0 xfail=0 skip=0 error=1 <<END
+oops
+1..2
+ok 1
+ok 2
+END
+
+err='test plan in middle of output'
+my_check total=3 pass=2 fail=0 xpass=0 xfail=0 skip=0 error=1 <<END
+ok 1
+ok 2
+1..2
+oops
+END
+
+# The two test plans here are deliberately equal.
+err='multiple test plans'
+my_check total=3 pass=2 fail=0 xpass=0 xfail=0 skip=0 error=1 <<END
+1..2
+ok 1
+ok 2
+1..2
+END
+
+err='missing test plan'
+my_check total=2 pass=1 fail=0 xpass=0 xfail=0 skip=0 error=1 <<END
+ok 1
+END
+
+:
diff --git a/tests/tap-plan.test b/tests/tap-plan.test
new file mode 100755
index 0000000..10fa377
--- /dev/null
+++ b/tests/tap-plan.test
@@ -0,0 +1,69 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support:
+# - test scripts with the test plan at the beginning
+# - test scripts with the test plan at the end
+# - test scripts without a test plan
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TEST_LOG_COMPILER = cat
+TESTS = top.test bot.test
+END
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+cat > top.test <<END
+1..3
+ok 1
+ok 2 # SKIP
+ok 3
+# a trailing comment
+END
+
+cat > bot.test <<END
+ok 1
+# a comment
+ok 2
+not ok 3 # TODO
+# another comment
+ok
+1..4
+END
+
+# Check that the plans doesn't cause any problem
+
+$MAKE check >stdout || { cat stdout; Exit 1; }
+cat stdout
+count_test_results total=7 pass=5 xfail=1 skip=1 fail=0 xpass=0 error=0
+
+:
diff --git a/tests/tap-realtime.test b/tests/tap-realtime.test
new file mode 100755
index 0000000..0f43e9e
--- /dev/null
+++ b/tests/tap-realtime.test
@@ -0,0 +1,115 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support:
+# - testsuite progress on console should happen mostly "in real time";
+# i.e., it's not acceptable for the driver to wait the end of the
+# script to start displaying results from it.
+# FIXME: this test uses expect(1) to ensure line buffering from make and
+# children, and is pretty hacky and complex; is there a better way to
+# accomplish the checks done here?
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cat >expect-check <<'END'
+eval spawn $env(SHELL) -c ":"
+expect eof
+END
+expect -f expect-check || {
+ echo "$me: failed to find a working expect program" >&2
+ Exit 77
+}
+rm -f expect-check
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TESTS = all.test
+END
+
+cat > all.test <<'END'
+#! /bin/sh
+echo 1..3
+
+# FIXME: creative quoting to placate maintainer-check
+sleep="sleep "3
+
+echo ok 1 - foo
+$sleep
+test -f ok-1 || { echo 'Bail out!'; exit 1; }
+
+echo ok 2 - bar
+$sleep
+test -f ok-2 || { echo 'Bail out!'; exit 1; }
+
+echo ok 3 - baz
+$sleep
+test -f ok-3 || { echo 'Bail out!'; exit 1; }
+
+: > all-is-well
+END
+
+chmod a+x all.test
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+cat > expect-make <<'END'
+eval spawn $env(MAKE) check
+expect {
+ "PASS: all.test 1 - foo" {
+ open "ok-1" "w"
+ exp_continue
+ }
+ "PASS: all.test 2 - bar" {
+ open "ok-2" "w"
+ exp_continue
+ }
+ "PASS: all.test 3 - baz" {
+ open "ok-3" "w"
+ exp_continue
+ }
+ "Testsuite summary" {
+ exit 0
+ }
+ timeout {
+ puts "expect timed out"
+ exit 1
+ }
+ default {
+ puts "expect error"
+ exit 1
+ }
+}
+END
+
+# Expect should simulate a tty as stdout, which should ensure a
+# line-buffered output.
+MAKE=$MAKE expect -f expect-make
+test -f all-is-well
+
+:
diff --git a/tests/tap-recheck-logs.test b/tests/tap-recheck-logs.test
new file mode 100755
index 0000000..a3484bb
--- /dev/null
+++ b/tests/tap-recheck-logs.test
@@ -0,0 +1,132 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support:
+# - RECHECK_LOGS
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TEST_LOG_COMPILER = cat
+TESTS = foo.test bar.test baz.test
+baz.log: zardoz
+END
+
+: > zardoz
+
+cat > foo.test <<'END'
+1..2
+ok 1
+ok 2
+END
+
+cat > bar.test <<'END'
+1..1
+not ok 1
+END
+
+cat > baz.test <<'END'
+1..1
+Bail out!
+END
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE -a
+
+./configure
+
+# Even the tests that are not re-run should contribute to the testsuite
+# summary when obtained by "make check RECHECK_LOGS=".
+grep_summary ()
+{
+ grep '^# TOTAL: *4$' stdout
+ grep '^# PASS: *2$' stdout
+ grep '^# XPASS: *0$' stdout
+ grep '^# FAIL: *1$' stdout
+ grep '^# XFAIL: *0$' stdout
+ grep '^# SKIP: *0$' stdout
+ grep '^# ERROR: *1$' stdout
+}
+
+$MAKE -e check && Exit 1
+test -f foo.log
+test -f bar.log
+test -f baz.log
+
+rm -f foo.log bar.log
+
+env RECHECK_LOGS= $MAKE -e check > stdout && { cat stdout; Exit 1; }
+cat stdout
+test -f foo.log
+test -f bar.log
+grep '^PASS: foo\.test 1$' stdout
+grep '^PASS: foo\.test 2$' stdout
+grep '^FAIL: bar\.test 1$' stdout
+grep 'baz\.test' stdout && Exit 1
+grep_summary
+
+$sleep
+touch foo.test
+# We re-run only a successful test, but the tests that failed in the
+# previous run should still be taken into account, and cause an overall
+# failure.
+env RECHECK_LOGS= $MAKE -e check > stdout && { cat stdout; Exit 1; }
+cat stdout
+grep '^PASS: foo\.test 1$' stdout
+grep '^PASS: foo\.test 2$' stdout
+grep 'ba[rz]\.test' stdout && Exit 1
+is_newest foo.log foo.test
+grep_summary
+
+$sleep
+touch zardoz
+env RECHECK_LOGS= $MAKE -e check > stdout && { cat stdout; Exit 1; }
+cat stdout
+grep '^ERROR: baz\.test' stdout
+$EGREP '(foo|bar)\.test' stdout && Exit 1
+is_newest baz.log zardoz
+grep_summary
+
+# Now, explicitly retry with all test logs already updated, and ensure
+# that the summary is still displayed.
+env RECHECK_LOGS= $MAKE -e check > stdout && { cat stdout; Exit 1; }
+cat stdout
+$EGREP '(foo|bar|baz)\.test' stdout && Exit 1
+grep_summary
+
+# The following should re-run foo.test (and only foo.test), even if its
+# log file is up-to-date.
+: > older
+env RECHECK_LOGS=foo.log $MAKE -e check > stdout && { cat stdout; Exit 1; }
+cat stdout
+grep '^PASS: foo\.test 1$' stdout
+grep '^PASS: foo\.test 2$' stdout
+grep 'ba[rz]\.test' stdout && Exit 1
+is_newest foo.log older
+grep_summary
+
+:
diff --git a/tests/tap-recheck.test b/tests/tap-recheck.test
new file mode 100755
index 0000000..14b02ad
--- /dev/null
+++ b/tests/tap-recheck.test
@@ -0,0 +1,217 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# Test the 'recheck' target for TAP test protocol.
+# Keep in sync with 'test-driver-custom-multitest-recheck.test'.
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TESTS = a.test b.test c.test d.test
+END
+
+cat > a.test << 'END'
+#! /bin/sh
+echo 1..2
+echo ok 1
+echo ok 2
+: > a.run
+END
+
+cat > b.test << 'END'
+#! /bin/sh
+: > b.run
+echo 1..2
+echo ok 1
+if test -f b.ok; then
+ echo ok 2
+else
+ echo 'Bail out!'
+fi
+END
+
+cat > c.test << 'END'
+#! /bin/sh
+echo 1..3
+if test -f c.pass; then
+ echo 'ok - c is ok :-)'
+else
+ echo 'not ok - c is ko :-('
+fi
+if test -f c.xfail; then
+ echo 'not ok 2 # TODO'
+else
+ echo 'ok 2 # TODO'
+fi
+echo 'not ok 3 - blah blah # TODO need better diagnostic'
+: > c.run
+END
+
+cat > d.test << 'END'
+#! /bin/sh
+test -f ./d.count && . ./d.count
+echo 1..${test_count-2}
+echo ok 1 '# SKIP: who cares ...'
+(. ./d.extra) || echo 'not ok 2 - d.extra failed'
+: > d.run
+END
+
+chmod a+x *.test
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+do_recheck ()
+{
+ case $* in
+ --fail) on_bad_rc='&&';;
+ --pass) on_bad_rc='||';;
+ *) fatal_ "invalid usage of function 'do_recheck'";;
+ esac
+ rm -f *.run
+ eval "\$MAKE recheck >stdout $on_bad_rc { cat stdout; ls -l; Exit 1; }; :"
+ cat stdout; ls -l
+}
+
+for vpath in : false; do
+ if $vpath; then
+ mkdir build
+ cd build
+ srcdir=..
+ else
+ srcdir=.
+ fi
+
+ $srcdir/configure
+
+ : A "make recheck" in a clean tree should run no tests.
+ do_recheck --pass
+ cat test-suite.log
+ test ! -r a.run
+ test ! -r a.log
+ test ! -r b.run
+ test ! -r b.log
+ test ! -r c.run
+ test ! -r c.log
+ test ! -r d.run
+ test ! -r d.log
+ count_test_results total=0 pass=0 fail=0 xpass=0 xfail=0 skip=0 error=0
+
+ : Run the tests for the first time.
+ $MAKE check >stdout && { cat stdout; Exit 1; }
+ cat stdout
+ ls -l
+ # All the test scripts should have run.
+ test -f a.run
+ test -f b.run
+ test -f c.run
+ test -f d.run
+ count_test_results total=9 pass=3 fail=2 xpass=1 xfail=1 skip=1 error=1
+
+ : Let us make b.test pass.
+ echo OK > b.ok
+ do_recheck --fail
+ # a.test has been successful the first time, so no need to re-run it.
+ # Similar considerations apply to similar checks, below.
+ test ! -r a.run
+ test -f b.run
+ test -f c.run
+ test -f d.run
+ count_test_results total=7 pass=2 fail=2 xpass=1 xfail=1 skip=1 error=0
+
+ : Let us make the first part of c.test pass.
+ echo OK > c.pass
+ do_recheck --fail
+ test ! -r a.run
+ test ! -r b.run
+ test -f c.run
+ test -f d.run
+ count_test_results total=5 pass=1 fail=1 xpass=1 xfail=1 skip=1 error=0
+
+ : Let us make also the second part of c.test pass.
+ echo KO > c.xfail
+ do_recheck --fail
+ test ! -r a.run
+ test ! -r b.run
+ test -f c.run
+ test -f d.run
+ count_test_results total=5 pass=1 fail=1 xpass=0 xfail=2 skip=1 error=0
+
+ : Nothing changed, so only d.test should be run.
+ for i in 1 2; do
+ do_recheck --fail
+ test ! -r a.run
+ test ! -r b.run
+ test ! -r c.run
+ test -f d.run
+ count_test_results total=2 pass=0 fail=1 xpass=0 xfail=0 skip=1 error=0
+ done
+
+ : Let us make d.test run more testcases, and experience _more_ failures.
+ echo 'test_count=9' > d.count
+ unindent > d.extra <<'END'
+ echo 'ok # SKIP s'
+ echo 'not ok - f 1'
+ echo 'ok - p 1'
+ echo 'not ok - f 2'
+ echo 'ok # TODO xp'
+ echo 'not ok - f 3'
+ echo 'not ok - f 4'
+ echo 'ok - p 2'
+ echo 'ok' # Extra test.
+ echo 'Bail out!'
+END
+ do_recheck --fail
+ test ! -r a.run
+ test ! -r b.run
+ test ! -r c.run
+ test -f d.run
+ count_test_results total=11 pass=2 fail=4 xpass=1 xfail=0 skip=2 error=2
+
+ : Let us finally make d.test pass.
+ echo 'test_count=1' > d.count
+ echo : > d.extra
+ do_recheck --pass
+ test ! -r a.run
+ test ! -r b.run
+ test ! -r c.run
+ test -f d.run
+ count_test_results total=1 pass=0 fail=0 xpass=0 xfail=0 skip=1 error=0
+
+ : All tests have been successful or skipped, nothing should be re-run.
+ do_recheck --pass
+ test ! -r a.run
+ test ! -r b.run
+ test ! -r c.run
+ test ! -r d.run
+ count_test_results total=0 pass=0 fail=0 xpass=0 xfail=0 skip=0 error=0
+
+ cd $srcdir
+
+done
+
+:
diff --git a/tests/tap-signal.test b/tests/tap-signal.test
new file mode 100755
index 0000000..b1ac2df
--- /dev/null
+++ b/tests/tap-signal.test
@@ -0,0 +1,65 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support:
+# - a test script terminated by a signal causes an hard error
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TESTS =
+END
+
+for sig in 1 2 13 15; do
+ unindent > signal-$sig.test <<END
+ #!/bin/sh
+ echo 1..1
+ echo ok 1
+ kill -$sig \$\$
+ echo "Bail out! \$0 not killed?"
+END
+ echo TESTS += signal-$sig.test >> Makefile.am
+done
+
+chmod a+x *.test
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+for append in '' 'TEST_LOG_DRIVER_FLAGS = --ignore-exit'; do
+ echo "$append" >> Makefile
+ $MAKE check >stdout && { cat stdout; Exit 1; }
+ cat stdout
+ count_test_results total=8 pass=4 fail=0 xpass=0 xfail=0 skip=0 error=4
+ for sig in 1 2 13 15; do
+ grep "^ERROR: signal-$sig\\.test - terminated by signal.*" stdout
+ done
+done
+
+:
diff --git a/tests/tap-skip-whole-whitespace.test
b/tests/tap-skip-whole-whitespace.test
new file mode 100755
index 0000000..2a19d4c
--- /dev/null
+++ b/tests/tap-skip-whole-whitespace.test
@@ -0,0 +1,68 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support:
+# - normalization of whitespace in console testsuite progress associated
+# with a SKIP directive in the TAP plan
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+sp=' '
+tab=' '
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TEST_LOG_COMPILER = cat
+TESTS = foo.test bar.test baz.test
+END
+
+cat > foo.test <<END
+1..0${tab}${tab} #${tab}SKIP ${tab}Strip leading & trailing ${tab}${tab}
+END
+
+cat > bar.test <<END
+1..0 #SKIP Preserve ${tab} middle${tab}${tab}space
+END
+
+cat > baz.test <<END
+1..0 # SKIP${tab} Strip${tab}external preserve ${tab}middle ${tab}${sp}
+END
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+$MAKE check > stdout || { cat stdout; Exit 1; }
+cat stdout
+
+count_test_results total=3 pass=0 fail=0 error=0 xpass=0 xfail=0 skip=3
+
+grep "SKIP: foo\\.test - Strip leading & trailing$" stdout
+grep "SKIP: bar\\.test - Preserve ${tab} middle${tab}${tab}space" stdout
+grep "SKIP: baz\\.test - Strip${tab}external preserve ${tab}middle" stdout
+
+:
diff --git a/tests/tap-skip-whole.test b/tests/tap-skip-whole.test
new file mode 100755
index 0000000..bd75e90
--- /dev/null
+++ b/tests/tap-skip-whole.test
@@ -0,0 +1,100 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# Basic TAP test protocol support:
+# - special plan format to skip all the tests in a script
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+weirdchars=\''"$!&()[]<>#;^?*/@%=,.:'
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TEST_LOG_COMPILER = cat
+TESTS = foo.test bar.test baz.test wget.test curl.test mu.test
+END
+
+cat > foo.test <<END
+1..0
+not ok 1
+not ok 2
+END
+
+cat > bar.test <<END
+1..0$tab $tab
+ok 1
+Bail out! some random failure
+END
+
+# It is undefined whether the comment after the plan below should
+# count as an explanation; still, the test should be skipped.
+cat > baz.test <<END
+1..0 # WWW::Mechanize not installed
+ok 1
+END
+
+cat > wget.test <<END
+1..0 # SKIP wget(1) not installed
+not ok 1 # TODO
+ok 2
+ok 3 # SKIP
+END
+
+cat > curl.test <<END
+1..0 # skip: Can't connect to gnu.org!
+not ok 1
+ok 2 # TODO
+END
+
+cat > mu.test <<END
+1..0 # Skip $weirdchars
+# Various TAP errors in here shouldn't be seen, as the test is skipped.
+1..2
+ok 1
+ok 22
+not ok 333
+1..7
+END
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+$MAKE check >stdout || { cat stdout; Exit 1; }
+cat stdout
+
+count_test_results total=6 pass=0 fail=0 xpass=0 xfail=0 skip=6 error=0
+
+grep '^SKIP: foo\.test$' stdout
+grep '^SKIP: bar\.test$' stdout
+grep '^SKIP: baz\.test' stdout # Deliberately laxer, see above for why.
+grep '^SKIP: wget\.test .* wget(1) not installed$' stdout
+grep '^SKIP: curl\.test .* Can'\''t connect to gnu\.org!$' stdout
+grep '^SKIP: mu\.test' stdout | $FGREP "$weirdchars" stdout
+test `grep -c ': .*\.test' stdout` -eq 6
+
+:
diff --git a/tests/tap-summary.test b/tests/tap-summary.test
new file mode 100755
index 0000000..649485e
--- /dev/null
+++ b/tests/tap-summary.test
@@ -0,0 +1,363 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support:
+# - testsuite summary
+# - colorized testsuite summary
+# TODO: splitting this into two distinct test scripts might be worthwhile
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat > configure.in <<END
+AC_INIT([GNU AutoTAP], [5.12], address@hidden)
+AM_INIT_AUTOMAKE([parallel-tests])
+AC_SUBST([PERL], ['$PERL'])
+AC_CONFIG_FILES([Makefile])
+AC_OUTPUT
+END
+
+cat > Makefile.stub << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TEST_LOG_COMPILER = cat
+TESTS = all.test
+END
+
+$ACLOCAL
+$AUTOCONF
+
+br='============================================================================'
+
+# The following shell variables are influential for this function:
+# - expect_failure
+# - use_colors
+do_check ()
+{
+ case $#,$1 in
+ 1,--pass) expect_failure=no;;
+ 1,--fail) expect_failure=yes;;
+ *) fatal_ "invalid usage of 'do_check'";;
+ esac
+ shift
+ cat > summary.exp
+ cat all.test
+ st=0
+ $MAKE check > stdout || st=$?
+ cat stdout
+ if test $expect_failure = yes; then
+ test $st -gt 0 || Exit 1
+ else
+ test $st -eq 0 || Exit 1
+ fi
+ $PERL -w "$testsrcdir"/extract-testsuite-summary stdout > summary.got \
+ || fatal_ "cannot extract testsuite summary"
+ cat summary.exp
+ cat summary.got
+ if test $use_colors = yes; then
+ # Use cmp, not diff, because the files might contain binary data.
+ compare=cmp
+ else
+ compare=diff
+ fi
+ $compare summary.exp summary.got || Exit 1
+}
+
+# FIXME: also enable testing of colors!
+#for use_colors in "no" "yes"; do
+for use_colors in "no"; do
+
+ cp Makefile.stub Makefile.am
+
+ if test $use_colors = yes; then
+ red='[0;31m'
+ grn='[0;32m'
+ lgn='[1;32m'
+ blu='[1;34m'
+ mgn='[0;35m'
+ brg='[1m'
+ std='[m'
+ echo AUTOMAKE_OPTIONS = color-tests >> Makefile.am
+ else
+ red= grn= lgn= blu= mgn= brg= std=
+ fi
+
+ success_header="\
+${grn}${br}${std}
+${grn}Testsuite summary for GNU AutoTAP 5.12${std}
+${grn}${br}${std}"
+
+ success_footer=${grn}${br}${std}
+
+ failure_header="\
+${red}${br}${std}
+${red}Testsuite summary for GNU AutoTAP 5.12${std}
+${red}${br}${std}"
+
+ failure_footer="\
+${red}${br}${std}
+${red}See ./test-suite.log${std}
+${red}Please report to address@hidden
+${red}${br}${std}"
+
+ $AUTOMAKE --add-missing
+
+ ./configure
+
+ # 1 pass.
+ { echo 1..1 && echo ok; } > all.test
+ do_check --pass <<END
+$success_header
+${brg}# TOTAL: 1${std}
+${grn}# PASS: 1${std}
+# SKIP: 0
+# XFAIL: 0
+# FAIL: 0
+# XPASS: 0
+# ERROR: 0
+$success_footer
+END
+
+ # 1 skip.
+ { echo 1..1 && echo 'ok # SKIP'; } > all.test
+ do_check --pass <<END
+$success_header
+${brg}# TOTAL: 1${std}
+# PASS: 0
+${blu}# SKIP: 1${std}
+# XFAIL: 0
+# FAIL: 0
+# XPASS: 0
+# ERROR: 0
+$success_footer
+END
+
+ # 1 xfail.
+ { echo 1..1 && echo 'not ok # TODO'; } > all.test
+ do_check --pass <<END
+$success_header
+${brg}# TOTAL: 1${std}
+# PASS: 0
+# SKIP: 0
+${lgn}# XFAIL: 1${std}
+# FAIL: 0
+# XPASS: 0
+# ERROR: 0
+$success_footer
+END
+
+ # 1 fail.
+ { echo 1..1 && echo not ok; } > all.test
+ do_check --fail <<END
+$failure_header
+${brg}# TOTAL: 1${std}
+# PASS: 0
+# SKIP: 0
+# XFAIL: 0
+${red}# FAIL: 1${std}
+# XPASS: 0
+# ERROR: 0
+$failure_footer
+END
+
+ # 1 xpass.
+ { echo 1..1 && echo 'ok # TODO'; } > all.test
+ do_check --fail <<END
+$failure_header
+${brg}# TOTAL: 1${std}
+# PASS: 0
+# SKIP: 0
+# XFAIL: 0
+# FAIL: 0
+${red}# XPASS: 1${std}
+# ERROR: 0
+$failure_footer
+END
+
+ # 1 hard error.
+ { echo 1..1 && echo 'Bail out!'; } > all.test
+ do_check --fail <<END
+$failure_header
+${brg}# TOTAL: 1${std}
+# PASS: 0
+# SKIP: 0
+# XFAIL: 0
+# FAIL: 0
+# XPASS: 0
+${mgn}# ERROR: 1${std}
+$failure_footer
+END
+
+ # 3 non-failing results.
+ cat > all.test <<END
+1..3
+ok
+not ok # TODO
+ok # SKIP
+END
+ do_check --pass <<END
+$success_header
+${brg}# TOTAL: 3${std}
+${grn}# PASS: 1${std}
+${blu}# SKIP: 1${std}
+${lgn}# XFAIL: 1${std}
+# FAIL: 0
+# XPASS: 0
+# ERROR: 0
+$success_footer
+END
+
+ # 1 pass, 1 skip, 1 fail.
+ cat > all.test <<END
+1..3
+ok
+ok # SKIP
+not ok
+END
+ do_check --fail <<END
+$failure_header
+${brg}# TOTAL: 3${std}
+${grn}# PASS: 1${std}
+${blu}# SKIP: 1${std}
+# XFAIL: 0
+${red}# FAIL: 1${std}
+# XPASS: 0
+# ERROR: 0
+$failure_footer
+END
+
+ # 1 pass, 1 xfail, 1 xpass.
+ cat > all.test <<END
+1..3
+ok
+ok # TODO
+not ok # TODO
+END
+ do_check --fail <<END
+$failure_header
+${brg}# TOTAL: 3${std}
+${grn}# PASS: 1${std}
+# SKIP: 0
+${lgn}# XFAIL: 1${std}
+# FAIL: 0
+${red}# XPASS: 1${std}
+# ERROR: 0
+$failure_footer
+END
+
+ # 1 skip, 1 xfail, 1 error.
+ cat > all.test <<END
+1..3
+ok # SKIP
+not ok # TODO
+Bail out!
+END
+ do_check --fail <<END
+$failure_header
+${brg}# TOTAL: 3${std}
+# PASS: 0
+${blu}# SKIP: 1${std}
+${lgn}# XFAIL: 1${std}
+# FAIL: 0
+# XPASS: 0
+${mgn}# ERROR: 1${std}
+$failure_footer
+END
+
+ # 1 of each kind
+ cat > all.test <<END
+1..6
+ok
+not ok
+ok # TODO
+not ok # TODO
+ok # SKIP
+Bail out!
+END
+ do_check --fail <<END
+$failure_header
+${brg}# TOTAL: 6${std}
+${grn}# PASS: 1${std}
+${blu}# SKIP: 1${std}
+${lgn}# XFAIL: 1${std}
+${red}# FAIL: 1${std}
+${red}# XPASS: 1${std}
+${mgn}# ERROR: 1${std}
+$failure_footer
+END
+
+ # Prepare some common data for later.
+ for i in 0 1 2 3 4 5 6 7 8 9; do
+ for j in 0 1 2 3 4 5 6 7 8 9; do
+ echo "ok"
+ echo "not ok # TODO"
+ echo "ok # SKIP"
+ done
+ done > tap
+
+ # Lots of non-failures (300 per kind).
+ (cat tap && cat tap && cat tap) > all.test
+ test `wc -l <all.test` -eq 900 || Exit 99 # Sanity check.
+ echo 1..900 >> all.test # Test plan.
+ do_check --pass <<END
+$success_header
+${brg}# TOTAL: 900${std}
+${grn}# PASS: 300${std}
+${blu}# SKIP: 300${std}
+${lgn}# XFAIL: 300${std}
+# FAIL: 0
+# XPASS: 0
+# ERROR: 0
+$success_footer
+END
+
+ # 1 failure and lots of non-failures means failure.
+ (cat tap && echo "not ok" && cat tap) > all.test
+ test `wc -l <all.test` -eq 601 || Exit 99 # Sanity check.
+ echo 1..601 >> all.test # Test plan.
+ do_check --fail <<END
+$failure_header
+${brg}# TOTAL: 601${std}
+${grn}# PASS: 200${std}
+${blu}# SKIP: 200${std}
+${lgn}# XFAIL: 200${std}
+${red}# FAIL: 1${std}
+# XPASS: 0
+# ERROR: 0
+$failure_footer
+END
+
+ # 1 error and lots of non-failures means failure.
+ (cat tap && sed 30q tap && echo 'Bail out!') > all.test
+ test `wc -l <all.test` -eq 331 || Exit 99 # Sanity check.
+ echo 1..331 >> all.test # Test plan.
+ do_check --fail <<END
+$failure_header
+${brg}# TOTAL: 331${std}
+${grn}# PASS: 110${std}
+${blu}# SKIP: 110${std}
+${lgn}# XFAIL: 110${std}
+# FAIL: 0
+# XPASS: 0
+${mgn}# ERROR: 1${std}
+$failure_footer
+END
+
+done # use_colors in "yes" "no"
+
+:
diff --git a/tests/tap-todo-skip-together.test
b/tests/tap-todo-skip-together.test
new file mode 100755
index 0000000..1528ca3
--- /dev/null
+++ b/tests/tap-todo-skip-together.test
@@ -0,0 +1,57 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support:
+# - TODO and SKIP directives on the same line: the first one wins
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TEST_LOG_COMPILER = cat
+TESTS = all.test
+END
+
+cat > all.test <<END
+1..2
+ok 1 # SKIP TODO
+not ok 2 # TODO SKIP
+END
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+$MAKE check >stdout || { cat stdout; Exit 1; }
+cat stdout
+
+count_test_results total=2 pass=0 fail=0 xpass=0 xfail=1 skip=1 error=0
+
+grep '^SKIP: all\.test 1 ' stdout
+grep '^XFAIL: all\.test 2 ' stdout
+
+:
diff --git a/tests/tap-todo-skip-whitespace.test
b/tests/tap-todo-skip-whitespace.test
new file mode 100755
index 0000000..68fbbc1
--- /dev/null
+++ b/tests/tap-todo-skip-whitespace.test
@@ -0,0 +1,118 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support:
+# - normalization of whitespace in console testsuite progress associated
+# with TODO and SKIP directives
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TEST_LOG_COMPILER = cat
+TESTS = all.test
+END
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+cat > stub.tap <<END
+1 # TODO
+2 # TODO foo?
+3 # TODO: bar!
+4 aa # TODO
+5 bb # TODO fnord 5
+6 cc # TODO:${tab}fnord 6
+7 - x # TODO
+8 - y # TODO fnord $tab 8
+9 - z # TODO: fnord 9 $tab
+10# TODO x0
+11$tab# TODO x1
+12 $tab$tab # TODO x2
+13 asd# TODO x3
+14 sad$tab# TODO x4
+15 das$tab$tab # TODO x5
+END
+
+cat > stub.exp <<END
+1 # TODO
+2 # TODO foo?
+3 # TODO: bar!
+4 aa # TODO
+5 bb # TODO fnord 5
+6 cc # TODO:${tab}fnord 6
+7 - x # TODO
+8 - y # TODO fnord $tab 8
+9 - z # TODO: fnord 9
+10 # TODO x0
+11 # TODO x1
+12 # TODO x2
+13 asd # TODO x3
+14 sad # TODO x4
+15 das # TODO x5
+END
+
+plan=1..15
+
+my_make_check ()
+{
+ xpass=0 xfail=0 skip=0
+ case $1 in
+ xpass|xfail|skip) eval $1=15;;
+ *) fatal_ "bad argument '$1' for my_make_check";;
+ esac
+ cat all.test
+ $MAKE check >stdout || : # Don't care about the exit status in this test.
+ cat stdout
+ count_test_results total=15 pass=0 fail=0 error=0 \
+ xpass=$xpass xfail=$xfail skip=$skip
+ # Don't be too strict w.r.t. possible normalization of "TODO: foo" into
+ # "TODO : foo" (as is done by, e.g., the 'TAP::Parser' perl module).
+ LC_ALL=C grep '^[A-Z][A-Z]*:' stdout \
+ | sed -e 's/# \(TODO\|SKIP\) *:/# \1:/' > got
+ cat exp
+ cat got
+ diff exp got
+}
+
+# For "TODO" directives leading to XPASS results.
+(echo $plan && sed -e 's/^/ok /' stub.tap) > all.test
+sed -e 's/^/XPASS: all.test /' stub.exp > exp
+my_make_check xpass
+
+# For "TODO" directives leading to XFAIL results.
+(echo $plan && sed -e 's/^/not ok /' stub.tap) > all.test
+sed -e 's/^/XFAIL: all.test /' stub.exp > exp
+my_make_check xfail
+
+# For "SKIP" directives.
+(echo $plan && sed -e 's/^/ok /' -e 's/TODO/SKIP/' stub.tap) > all.test
+sed -e 's/TODO/SKIP/' -e 's/^/SKIP: all.test /' stub.exp > exp
+my_make_check skip
+
+:
diff --git a/tests/tap-todo-skip.test b/tests/tap-todo-skip.test
new file mode 100755
index 0000000..c37b631
--- /dev/null
+++ b/tests/tap-todo-skip.test
@@ -0,0 +1,109 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support:
+# - TODO and SKIP directives are case-insensitive
+# - TODO and SKIP directives can be followed optionally by a colon ":"
+# and by an optional explanation.
+# - our driver isn't fooled into recognizing TODO and SKIP directives
+# spuriously
+# - the reasons for TODO and SKIP, if present, are nicely printed in
+# the testsuite progress output
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TEST_LOG_COMPILER = cat
+TESTS = all.test
+END
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+# ----------------------------------------------------- #
+# Check all possible combinations of: #
+# - uppercase/lowercase #
+# - with/without colon character ":" #
+# - with/without explanatory message #
+# in TODO and SKIP directives. #
+# ----------------------------------------------------- #
+
+# There are 2 * 2^6 + 2 * 2^6 = 256 tests.
+echo 1..256 > all.test
+
+# These nested loops are clearer without indentation.
+
+for c1 in t T; do
+for c2 in o O; do
+for c3 in d D; do
+for c4 in o O; do
+for ex in '' ':' ' foo' ': foo'; do
+ echo "not ok # $c1$c2$c3$c4$ex"
+ echo "not ok# $c1$c2$c3$c4$ex"
+done; done; done; done; done >> all.test
+
+for c1 in s S; do
+for c2 in k K; do
+for c3 in i I; do
+for c4 in p P; do
+for ex in '' ':' ' foo' ': foo'; do
+ echo "ok # $c1$c2$c3$c4$ex"
+ echo "ok# $c1$c2$c3$c4$ex"
+done; done; done; done; done >> all.test
+
+cat all.test # For debugging.
+
+$MAKE check >stdout || { cat stdout; Exit 1; }
+cat stdout
+
+count_test_results total=256 pass=0 fail=0 xpass=0 xfail=128 skip=128 error=0
+
+# -------------------------------------------------------- #
+# TODO ans SKIP directives aren't recognized spuriously. #
+# -------------------------------------------------------- #
+
+cat > all.test <<'END'
+1..9
+ok TODO
+ok - TODO
+ok 3 TODO
+ok 4 - TODO
+ok SKIP
+ok - SKIP
+ok 7 SKIP
+ok 8 - SKIP
+ok 9
+END
+
+$MAKE check >stdout || { cat stdout; Exit 1; }
+cat stdout
+
+count_test_results total=9 pass=9 fail=0 xpass=0 xfail=0 skip=0 error=0
+
+:
diff --git a/tests/tap-unplanned.test b/tests/tap-unplanned.test
new file mode 100755
index 0000000..696f73b
--- /dev/null
+++ b/tests/tap-unplanned.test
@@ -0,0 +1,177 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support:
+# - unplanned tests are properly reported as errors
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TEST_LOG_COMPILER = cat
+TESTS = foo.test
+END
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+cat > foo.test <<END
+1..1
+ok 1
+ok 2
+END
+$MAKE check >stdout && { cat stdout; Exit 1; }
+cat stdout
+count_test_results total=3 pass=1 fail=0 xpass=0 xfail=0 skip=0 error=2
+grep '^ERROR: foo\.test - too many tests run (expected 1, got 2)$' stdout
+grep '^ERROR: foo\.test 2 # UNPLANNED$' stdout
+
+cat > foo.test <<END
+1..2
+ok 1
+ok 2
+ok 3
+END
+$MAKE check >stdout && { cat stdout; Exit 1; }
+cat stdout
+count_test_results total=4 pass=2 fail=0 xpass=0 xfail=0 skip=0 error=2
+grep '^ERROR: foo\.test - too many tests run (expected 2, got 3)$' stdout
+grep '^ERROR: foo\.test 3 # UNPLANNED$' stdout
+
+# Interaction with XFAIL_TESTS.
+cat > foo.test <<END
+1..2
+not ok 1
+ok 2 # SKIP
+ok 3
+not ok 4
+ok 5 # SKIP
+END
+XFAIL_TESTS=foo.test $MAKE -e check >stdout && { cat stdout; Exit 1; }
+cat stdout
+count_test_results total=6 pass=0 fail=0 xpass=0 xfail=1 skip=1 error=4
+grep '^ERROR: foo\.test - too many tests run (expected 2, got 5)$' stdout
+grep '^ERROR: foo\.test 3 # UNPLANNED$' stdout
+grep '^ERROR: foo\.test 4 # UNPLANNED$' stdout
+grep '^ERROR: foo\.test 5 # UNPLANNED$' stdout
+
+cat > foo.test <<END
+1..1
+ok 1
+
+ok
+ok 3
+ok foo
+ok 5 - bar bar
+
+not ok
+not ok 7
+not ok foo
+not ok 9 - bar bar
+
+ok # TODO
+ok 11 # TODO
+ok foo # TODO
+ok 13 - bar bar # TODO
+
+not ok # TODO
+not ok 15 # TODO
+not ok foo # TODO
+not ok 17 - bar bar # TODO
+
+ok # SKIP
+ok 19 # SKIP
+ok foo # SKIP
+ok 21 - bar bar # SKIP
+END
+
+cat > t <<END
+
+2
+3
+4 foo
+5 - bar bar
+
+6
+7
+8 foo
+9 - bar bar
+
+10
+11
+12 foo
+13 - bar bar
+
+14
+15
+16 foo
+17 - bar bar
+
+18
+19
+20 foo
+21 - bar bar
+
+END
+
+$MAKE check >stdout && { cat stdout; Exit 1; }
+cat stdout
+count_test_results total=22 pass=1 fail=0 xpass=0 xfail=0 skip=0 error=21
+
+echo 'PASS: foo.test 1' > exp
+sed -e '/^$/d' -e 's/.*/ERROR: foo.test & # UNPLANNED/' t >> exp
+echo 'ERROR: foo.test - too many tests run (expected 1, got 21)' >> exp
+
+$FGREP ': foo.test' stdout > got
+
+cat exp
+cat got
+diff exp got
+
+# Note that, if the TAP input has a trailing plan, it is not possible
+# to flag unplanned tests as such, since we do not know they're unplanned
+# until the plan is reached; still, we should give at least an error
+# message about the unmatched number of tests once we've got the plan.
+
+for x in 'ok' 'ok 3' 'not ok' 'not ok # TODO' 'ok # TODO' 'ok # SKIP'; do
+ unindent > foo.test <<END
+ ok 1
+ ok 2 # SKIP
+ $x
+ 1..2
+END
+ $MAKE check >stdout && { cat stdout; Exit 1; }
+ cat stdout
+ test `$FGREP -c ': foo.test' stdout` -eq 4
+ $EGREP '^PASS: foo\.test 1($| )' stdout
+ $EGREP '^SKIP: foo\.test 2($| )' stdout
+ $EGREP ': foo\.test 3($| )' stdout
+ grep '^ERROR: foo\.test - too many tests run (expected 2, got 3)$' stdout
+done
+
+:
diff --git a/tests/tap-whitespace-normalization.test
b/tests/tap-whitespace-normalization.test
new file mode 100755
index 0000000..16e6801
--- /dev/null
+++ b/tests/tap-whitespace-normalization.test
@@ -0,0 +1,181 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support: whitespace normalization (or lack thereof) in the testsuite
+# progress output on console. We keep all these checks here in a single
+# script so that a potential cosmetic change in the output format won't
+# force us to tweak dozens of other tests (hopefully).
+# See also related test 'tap-todo-skip-whitespace.test'.
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TEST_LOG_COMPILER = cat
+TESTS =
+END
+
+: > exp
+
+sp=' '
+tab=' '
+
+spaces_a=${sp}${tab}${tab}${sp}${sp}${tab}
+spaces_b=${tab}${tab}${sp}${tab}${sp}${sp}${sp}
+
+#-----------------------------------------------------------------------
+
+echo TESTS += numbers.test >> Makefile.am
+
+cat > numbers.test <<END
+1..6
+ok${spaces_a}1
+ok 2${spaces_b}
+ok${spaces_a}3${spaces_b}
+not ok${spaces_b}4
+not ok 5${spaces_a}
+not ok${spaces_b}6${spaces_a}
+END
+
+cat >> exp <<END
+PASS: numbers.test 1
+PASS: numbers.test 2
+PASS: numbers.test 3
+FAIL: numbers.test 4
+FAIL: numbers.test 5
+FAIL: numbers.test 6
+END
+
+#-----------------------------------------------------------------------
+
+echo TESTS += description.test >> Makefile.am
+
+cat > description.test <<END
+1..8
+ok${spaces_a}+foo
+ok +bar${spaces_b}
+ok${spaces_a}+baz${spaces_b}
+not ok${spaces_b}-foo
+not ok -bar${spaces_a}
+not ok${spaces_b}-baz${spaces_a}
+ok u${spaces_b}v${spaces_a}w${sp}
+not ok${spaces_a}x${spaces_a}y${tab}z${tab}
+END
+
+cat >> exp <<END
+PASS: description.test 1 +foo
+PASS: description.test 2 +bar
+PASS: description.test 3 +baz
+FAIL: description.test 4 -foo
+FAIL: description.test 5 -bar
+FAIL: description.test 6 -baz
+PASS: description.test 7 u${spaces_b}v${spaces_a}w
+FAIL: description.test 8 x${spaces_a}y${tab}z
+END
+
+#-----------------------------------------------------------------------
+
+# "Bail out!" magic.
+
+echo TESTS += bailout.test >> Makefile.am
+
+cat > bailout.test <<END
+1..1
+Bail out!${tab}${sp}${sp}${tab}We're out of disk space.
+ok 1
+END
+
+cat >> exp <<END
+ERROR: bailout.test - Bail out! We're out of disk space.
+END
+
+echo TESTS += bailout2.test >> Makefile.am
+
+cat > bailout2.test <<END
+1..1
+Bail out!foo${tab}${sp}
+ok 1
+END
+
+cat >> exp <<END
+ERROR: bailout2.test - Bail out! foo
+END
+
+#-----------------------------------------------------------------------
+
+# Diagnostic lines.
+
+echo AM_TEST_LOG_DRIVER_FLAGS = --comments >> Makefile.am
+echo TESTS += cmnt.test >> Makefile.am
+
+cat > cmnt.test <<END
+1..1
+ok 1
+#Leading whitespace gets added
+# ${tab}${tab} ${tab}Extra leading whitespace is stripped
+# Trailing whitespace is stripped ${tab} ${tab}${tab}
+# Middle${tab}whitespace is${tab} ${tab}${tab} kept
+# ${tab} And only${tab}middle ${tab}whitespace ${tab}${tab} ${tab}
+END
+
+cat >> exp <<END
+PASS: cmnt.test 1
+# cmnt.test: Leading whitespace gets added
+# cmnt.test: Extra leading whitespace is stripped
+# cmnt.test: Trailing whitespace is stripped
+# cmnt.test: Middle${tab}whitespace is${tab} ${tab}${tab} kept
+# cmnt.test: And only${tab}middle ${tab}whitespace
+END
+
+#-----------------------------------------------------------------------
+
+# TODO: we should have more checks here ... (but let's not over-do FTM).
+
+#-----------------------------------------------------------------------
+
+chmod a+x *.test
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+# Don't care about exit status or number of test results, they should be
+# checked for in many other tests.
+$MAKE check >stdout || :
+cat stdout
+
+LC_ALL=C sort exp > t
+mv -f t exp
+
+# We need the sort below to account for parallel make usage.
+LC_ALL=C grep '[a-z0-9][a-z0-9]*\.test' stdout | LC_ALL=C sort > got
+
+cat exp
+cat got
+diff exp got
+
+:
diff --git a/tests/tap-with-and-without-number.test
b/tests/tap-with-and-without-number.test
new file mode 100755
index 0000000..a1f7a4b
--- /dev/null
+++ b/tests/tap-with-and-without-number.test
@@ -0,0 +1,68 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support:
+# - numbered and unnumbered test results can coexist
+# - tests without explicit number get automatically numbered in the
+# testsuite progress output on console
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TEST_LOG_COMPILER = cat
+TESTS = all.test
+END
+
+cat > all.test <<'END'
+1..7
+ok 1 foo
+ok 2 - foo2
+ok - bar
+not ok baz # TODO
+not ok 5 - asd # TODO
+ok 6 # SKIP
+ok zardoz
+END
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+$MAKE check >stdout || { cat stdout; Exit 1; }
+cat stdout
+count_test_results total=7 pass=4 fail=0 xpass=0 xfail=2 skip=1 error=0
+
+grep '^PASS: all\.test 1 foo$' stdout
+grep '^PASS: all\.test 2 - foo2$' stdout
+grep '^PASS: all\.test 3 - bar$' stdout
+grep '^XFAIL: all\.test 4 baz # TODO$' stdout
+grep '^XFAIL: all\.test 5 - asd # TODO$' stdout
+grep '^SKIP: all\.test 6 # SKIP$' stdout
+grep '^PASS: all\.test 7 zardoz$' stdout
+
+:
diff --git a/tests/tap-xfail-tests.test b/tests/tap-xfail-tests.test
new file mode 100755
index 0000000..f159e8a
--- /dev/null
+++ b/tests/tap-xfail-tests.test
@@ -0,0 +1,85 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP and $(XFAIL_TESTS): test results without directives are turned from
+# PASS to XPASS and from FAIL to XFAIL; other results are unchanged.
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TEST_LOG_COMPILER = cat
+TESTS = the.test
+XFAIL_TESTS = $(TESTS)
+END
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+cat > the.test <<END
+1..6
+ok 1
+not ok 2
+ok 3 # TODO
+not ok 4 # TODO
+ok 5 # SKIP
+# The next should count as a failure (and thus will be
+# rendered into an expected failure by XFAIL_TESTS).
+not ok 6 # SKIP
+Bail out!
+END
+
+$MAKE check >stdout && { cat stdout; Exit 1; }
+cat stdout
+
+count_test_results total=7 pass=0 fail=0 xpass=2 xfail=3 skip=1 error=1
+
+grep '^XPASS: the\.test 1$' stdout
+grep '^XFAIL: the\.test 2$' stdout
+grep '^XPASS: the\.test 3 # TODO' stdout
+grep '^XFAIL: the\.test 4 # TODO' stdout
+grep '^SKIP: the\.test 5 # SKIP' stdout
+grep '^XFAIL: the\.test 6 # SKIP' stdout
+grep '^ERROR: the\.test - Bail out!' stdout
+
+# Check that the exit status of the testsuite is indeed 0 when we
+# would expect success.
+
+cat > the.test <<END
+1..3
+not ok 1
+ok 2 # SKIP
+not ok 3 # TODO
+END
+
+$MAKE check >stdout || { cat stdout; Exit 1; }
+cat stdout
+
+count_test_results total=3 pass=0 fail=0 xpass=0 xfail=2 skip=1 error=0
+
+:
diff --git a/tests/test-driver-custom-multitest-recheck.test
b/tests/test-driver-custom-multitest-recheck.test
index 137ff96..8bbcc7c 100755
--- a/tests/test-driver-custom-multitest-recheck.test
+++ b/tests/test-driver-custom-multitest-recheck.test
@@ -20,6 +20,7 @@
# support, but also serves as a "usability test" for our APIs.
# See also related tests `test-driver-custom-multitest-recheck2.test'
# and `parallel-tests-recheck-override.test'.
+# Keep in sync with 'tap-recheck.test'.
parallel_tests=yes
. ./defs || Exit 1
--
1.7.2.3
- Re: [PATCH 1/2] test harness: allow more metadata in log files, (continued)
[PATCH 3/5] {test-protocols} parallel-tests: simplify testsuite summary, Stefano Lattarini, 2011/07/14
[PATCH 4/5] {test-protocols} tests defs: new auxiliary function 'count_test_results', Stefano Lattarini, 2011/07/14
[PATCH 5/5] {test-protocols} tap: add experimental TAP-aware driver,
Stefano Lattarini <=
Re: [GSoC] Some patches for testsuite harness improvements and TAP support introduction, Ralf Wildenhues, 2011/07/18
[PATCH 6/5] tap: some preparatory refactoring (1), Stefano Lattarini, 2011/07/18
[PATCH 7/5] tap: some preparatory refactoring (2), Stefano Lattarini, 2011/07/18
[PATCH 8/5] tap: support colorization of testsuite progress output, Stefano Lattarini, 2011/07/18