[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [GSoC] preliminary tests on TAP support
|
From: |
Stefano Lattarini |
|
Subject: |
Re: [GSoC] preliminary tests on TAP support |
|
Date: |
Mon, 11 Jul 2011 23:25:34 +0200 |
|
User-agent: |
KMail/1.13.3 (Linux/2.6.30-2-686; KDE/4.4.4; i686; ; ) |
On Saturday 09 July 2011, Stefano Lattarini wrote:
> On Thursday 07 July 2011, Stefano Lattarini wrote:
> > Hello Automakers.
> >
> > I've created a nice battery of tests on TAP support, and some sketchy
> > documentation. Some tests are still incomplete, but I think it's better
> > to start getting some early feedback anyway. Thus I've already pushed
> > them to the temporary branch 'GSoC/experimental/test-results-work' (note
> > that, since that branch is temporary and rewindable, the patch can still
> > be amended freely). Attached is what I've pushed.
> >
> > As usual, reviews, suggestions and criticism are welcome. I should be
> > able to finish the incomplete tests by tomorrow evening.
> >
> > Regards,
> > Stefano
> >
> I've now squashed in the attached patch. Notice that some tests are
> still incomplete or buggy, and there are even some incompatibilities
> between different tests; but these issues can be sorted out once we
> have the experimental/exploratory implementation ready and rolling.
>
> Regards,
> Stefano
>
And now here it is the first "working" version of the TAP driver (it is still
incomplete and slightly buggy, but makes lots of the new tests pass). Already
pushed to the temporary rewindable branch 'GSoC/experimental/test-results-work'.
Comments welcome!
Regards,
Stefano
From 17b1fce52099a76111920ed209157880605fefae Mon Sep 17 00:00:00 2001
Message-Id: <address@hidden>
From: Stefano Lattarini <address@hidden>
Date: Sun, 3 Jul 2011 19:20:22 +0200
Subject: [PATCH] tap: add experimental TAP-aware driver
---
ChangeLog | 47 +++
doc/automake.texi | 94 ++++++-
lib/Makefile.am | 2 +-
lib/Makefile.in | 2 +-
lib/tap-driver | 279 +++++++++++++++++
tests/Makefile.am | 37 +++
tests/Makefile.in | 37 +++
tests/tap-autonumber.test | 94 ++++++
tests/tap-bad-prog.test | 69 +++++
tests/tap-bailout.test | 155 ++++++++++
tests/tap-basic.test | 177 +++++++++++
tests/tap-color.test | 145 +++++++++
tests/tap-deps.test | 98 ++++++
tests/tap-diagnostic.test | 142 +++++++++
tests/tap-empty-diagnostic.test | 59 ++++
tests/tap-escape-directive.test | 57 ++++
tests/tap-exit.test | 76 +++++
tests/tap-fancy.test | 151 ++++++++++
tests/tap-fancy2.test | 147 +++++++++
tests/tap-global-result.test | 128 ++++++++
tests/tap-html.test | 93 ++++++
tests/tap-log.test | 174 +++++++++++
tests/tap-merge-stdout-stderr.test | 85 ++++++
tests/tap-more.test | 160 ++++++++++
tests/tap-more2.test | 93 ++++++
tests/tap-no-merge-stdout-stderr.test | 66 ++++
tests/tap-no-spurious-summary.test | 65 ++++
tests/tap-no-spurious.test | 114 +++++++
tests/tap-not-ok-skip.test | 57 ++++
tests/tap-numeric-description.test | 86 ++++++
tests/tap-out-of-order.test | 94 ++++++
tests/tap-passthrough.test | 188 ++++++++++++
tests/tap-plan-errors.test | 168 +++++++++++
tests/tap-plan.test | 69 +++++
tests/tap-realtime.test | 96 ++++++
tests/tap-recheck-logs.test | 132 ++++++++
tests/tap-recheck.test | 215 ++++++++++++++
tests/tap-skip-whole.test | 100 +++++++
tests/tap-summary.test | 363 +++++++++++++++++++++++
tests/tap-todo-skip-together.test | 57 ++++
tests/tap-todo-skip.test | 177 +++++++++++
tests/tap-whitespace-normalization.test | 179 +++++++++++
tests/tap-with-and-without-number.test | 58 ++++
tests/tap-xfail-tests.test | 87 ++++++
tests/test-driver-custom-multitest-recheck.test | 1 +
45 files changed, 4965 insertions(+), 8 deletions(-)
create mode 100755 lib/tap-driver
create mode 100755 tests/tap-autonumber.test
create mode 100755 tests/tap-bad-prog.test
create mode 100755 tests/tap-bailout.test
create mode 100755 tests/tap-basic.test
create mode 100755 tests/tap-color.test
create mode 100755 tests/tap-deps.test
create mode 100755 tests/tap-diagnostic.test
create mode 100755 tests/tap-empty-diagnostic.test
create mode 100755 tests/tap-escape-directive.test
create mode 100755 tests/tap-exit.test
create mode 100755 tests/tap-fancy.test
create mode 100755 tests/tap-fancy2.test
create mode 100755 tests/tap-global-result.test
create mode 100755 tests/tap-html.test
create mode 100755 tests/tap-log.test
create mode 100755 tests/tap-merge-stdout-stderr.test
create mode 100755 tests/tap-more.test
create mode 100755 tests/tap-more2.test
create mode 100755 tests/tap-no-merge-stdout-stderr.test
create mode 100755 tests/tap-no-spurious-summary.test
create mode 100755 tests/tap-no-spurious.test
create mode 100755 tests/tap-not-ok-skip.test
create mode 100755 tests/tap-numeric-description.test
create mode 100755 tests/tap-out-of-order.test
create mode 100755 tests/tap-passthrough.test
create mode 100755 tests/tap-plan-errors.test
create mode 100755 tests/tap-plan.test
create mode 100755 tests/tap-realtime.test
create mode 100755 tests/tap-recheck-logs.test
create mode 100755 tests/tap-recheck.test
create mode 100755 tests/tap-skip-whole.test
create mode 100755 tests/tap-summary.test
create mode 100755 tests/tap-todo-skip-together.test
create mode 100755 tests/tap-todo-skip.test
create mode 100755 tests/tap-whitespace-normalization.test
create mode 100755 tests/tap-with-and-without-number.test
create mode 100755 tests/tap-xfail-tests.test
diff --git a/ChangeLog b/ChangeLog
index d76b071..4a9d9c3 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,50 @@
+2011-07-11 Stefano Lattarini <address@hidden>
+
+ tap: add experimental TAP-aware driver
+ * doc/automake.texi (Using the TAP test protocol): New section.
+ (Overview of Custom Test Drivers Support): Minor updates.
+ * lib/tap-driver: New script, TAP-aware test driver for Automake.
+ Implemented in perl and based on TAP::Parser.
+ * lib/Makefile.am (dist_script_DATA): Add it.
+ * tests/tap-autonumber.test: New test.
+ * tests/tap-bad-prog.test: Likewise.
+ * tests/tap-bailout.test: Likewise.
+ * tests/tap-basic.test: Likewise.
+ * tests/tap-color.test: Likewise.
+ * tests/tap-deps.test: Likewise.
+ * tests/tap-diagnostic.test: Likewise.
+ * tests/tap-empty-diagnostic.test: Likewise.
+ * tests/tap-escape-directive.test: Likewise.
+ * tests/tap-exit.test: Likewise.
+ * tests/tap-fancy.test: Likewise.
+ * tests/tap-fancy2.test: Likewise.
+ * tests/tap-global-result.test: Likewise.
+ * tests/tap-html.test: Likewise.
+ * tests/tap-log.test: Likewise.
+ * tests/tap-merge-stdout-stderr.test: Likewise.
+ * tests/tap-more.test: Likewise.
+ * tests/tap-more2.test: Likewise.
+ * tests/tap-no-merge-stdout-stderr.test: Likewise.
+ * tests/tap-no-spurious-summary.test: Likewise.
+ * tests/tap-no-spurious.test: Likewise.
+ * tests/tap-not-ok-skip.test: Likewise.
+ * tests/tap-numeric-description.test: Likewise.
+ * tests/tap-out-of-order.test: Likewise.
+ * tests/tap-passthrough.test: Likewise.
+ * tests/tap-plan-errors.test: Likewise.
+ * tests/tap-plan.test: Likewise.
+ * tests/tap-realtime.test: Likewise.
+ * tests/tap-recheck-logs.test: Likewise.
+ * tests/tap-recheck.test: Likewise.
+ * tests/tap-skip-whole.test: Likewise.
+ * tests/tap-summary.test: Likewise.
+ * tests/tap-todo-skip-together.test: Likewise.
+ * tests/tap-todo-skip.test: Likewise.
+ * tests/tap-whitespace-normalization.test: Likewise.
+ * tests/tap-with-and-without-number.test: Likewise.
+ * tests/tap-xfail-tests.test: Likewise.
+ * tests/Makefile.am (TESTS): Update.
+
2011-07-07 Stefano Lattarini <address@hidden>
tests defs: new auxiliary function 'count_test_results'
diff --git a/doc/automake.texi b/doc/automake.texi
index 16804cd..0c75a22 100644
--- a/doc/automake.texi
+++ b/doc/automake.texi
@@ -320,6 +320,7 @@ Support for test suites
* Simple Tests:: Listing programs and scripts in @code{TESTS}
* Simple Tests using parallel-tests:: More powerful test driver
* Custom Test Drivers:: Writing and using custom test drivers
+* Using the TAP test protocol:: Integrating test scripts that use the TAP
protocol
* DejaGnu Tests:: Interfacing with the external testing framework
* Install Tests:: Running tests on installed packages
@@ -8656,6 +8657,7 @@ are very similar.
* Simple Tests:: Listing programs and scripts in @code{TESTS}
* Simple Tests using parallel-tests:: More powerful test driver
* Custom Test Drivers:: Writing and using custom test drivers
+* Using the TAP test protocol:: Integrating test scripts that use the TAP
protocol
* DejaGnu Tests:: Interfacing with the external testing framework
* Install Tests:: Running tests on installed packages
@end menu
@@ -9037,12 +9039,9 @@ or work around the issue with inference rules to
generate the tests.
@subsection Overview of Custom Test Drivers Support
Starting form Automake version 1.12, the @option{parallel-tests} harness
-allows the package authors to use third-party custom test drivers, in case
-the
address@hidden FIXME: this should become "default ones" once we have TAP and
Subunit
-default one is inadequate for their purposes.
address@hidden FIXME: add this once we have TAP and Subunit
address@hidden "or do not support their testing protocol of choice."
+allows the package authors to use third-party custom test drivers, in
+case the default ones are inadequate for their purposes, or do not support
+their testing protocol of choice.
A custom test driver is expected to properly run the test programs
passed to it, including the command-line arguments passed to it,
@@ -9336,6 +9335,89 @@ is not the case, the HTML generation will not work,
although all the
other functionalities of the Automake testsuite harness should remain
untouched, and continue to work correctly.
address@hidden Using the TAP test protocol
address@hidden Using the TAP test protocol
+
+Brief introduction to TAP and its philosophy.
+
+Real-word uses of TAP: testsuite of @command{perl} and of many
+perl modules, and of @command{git}. Third-party libraries and
+utilities that can generate TAP (tell also to look at links
+below).
+
address@hidden
+A harness must only read TAP output from standard output and not
+from standard error. Lines written to standard output matching
address@hidden/^(not )?ok\b/} must be interpreted as test lines. All other
+lines must not be considered test output.
address@hidden quotation
+
address@hidden
+Here are some links to more extensive official or third-party documentation
+and resources:
address@hidden @bullet
address@hidden
address@hidden://search.cpan.org/~petdance/Test-Harness/lib/Test/Harness/TAP.pod,
+ @samp{Test::Harness::TAP}},
+the (mostly) official documentation about the TAP format and protocol.
address@hidden
address@hidden://search.cpan.org/~andya/Test-Harness/bin/prove,
+ @command{prove}},
+the most famous command-line TAP test driver, included in the distribution
+of @command{perl} and
address@hidden://search.cpan.org/~andya/Test-Harness/lib/Test/Harness.pm,
+ @samp{Test::Harness}}.
address@hidden
+The @uref{http://testanything.org/wiki/index.php/Main_Page,TAP wiki}.
address@hidden
+A ``gentle introduction'' to testing for perl coders:
address@hidden://search.cpan.org/dist/Test-Simple/lib/Test/Tutorial.pod,
+ @samp{Test::Tutorial}}.
address@hidden
address@hidden://search.cpan.org/~mschwern/Test-Simple/lib/Test/Simple.pm,
+ @samp{Test::Simple}}
+and
address@hidden://search.cpan.org/~mschwern/Test-Simple/lib/Test/More.pm,
+ @samp{Test::More}},
+the standard perl testing libraries, which are based on TAP.
address@hidden
address@hidden://www.eyrie.org/~eagle/software/c-tap-harness/,C TAP Harness},
+a C-based project implementing both a TAP producer and a TAP consumer.
address@hidden itemize
+
+Give example of output from @command{prove} command (when used in
+``verbose'' mode, with option @option{-v}), to which we'd like to
+remain somewhat compatible compatible, while also (and foremost)
+being compatible with the default @option{parallel-tests} driver.
+
+Example of output from out own TAP testsuite driver. Point out
+similarities with the output from @command{prove --verbose} and
+from the default @option{parallel-tests} driver.
+
+Option @option{--ignore-exit} causes the driver to ignore the exit
+status of the test scripts; by default, the driver will report an
+error if the script exit with status != 0.
address@hidden: also add a @option{--no-ignore-exit} option, for
+completeness?
+
+By default, TAP diagnostic (i.e., lines beginning with the @samp{#}
+character) are copied only in the @file{.log} file. The option
address@hidden causes the driver to display them in the testsuite
+progress output too (@emph{TODO}: give example). The option
address@hidden restore the default behaviour.
+
address@hidden
+Differences and incompatibilities with other TAP parsers and drivers:
address@hidden @bullet
address@hidden
+A @code{Bail out!} directive doesn't stop the whole testsuite, but only
+the test script it occurs into. This doesn't follows TAP specifications,
+but on the other hand maximize compatibility (and code sharing) with
+the ``hard error'' concept of the default @option{parallel-tests} driver.
address@hidden
address@hidden: there's surely something else ...
address@hidden itemize
+
@node DejaGnu Tests
@section DejaGnu Tests
diff --git a/lib/Makefile.am b/lib/Makefile.am
index a26d663..a9467f1 100644
--- a/lib/Makefile.am
+++ b/lib/Makefile.am
@@ -29,7 +29,7 @@ dist_pkgvdata_DATA = COPYING INSTALL texinfo.tex ansi2knr.c
ansi2knr.1 \
scriptdir = $(pkgvdatadir)
dist_script_DATA = config.guess config.sub install-sh mdate-sh missing \
mkinstalldirs elisp-comp ylwrap acinstall depcomp compile py-compile \
- symlink-tree ar-lib test-driver
+ symlink-tree ar-lib test-driver tap-driver
EXTRA_DIST = gnupload
diff --git a/lib/Makefile.in b/lib/Makefile.in
index 5f8aa91..b600402 100644
--- a/lib/Makefile.in
+++ b/lib/Makefile.in
@@ -242,7 +242,7 @@ dist_pkgvdata_DATA = COPYING INSTALL texinfo.tex ansi2knr.c
ansi2knr.1 \
scriptdir = $(pkgvdatadir)
dist_script_DATA = config.guess config.sub install-sh mdate-sh missing \
mkinstalldirs elisp-comp ylwrap acinstall depcomp compile py-compile \
- symlink-tree ar-lib test-driver
+ symlink-tree ar-lib test-driver tap-driver
EXTRA_DIST = gnupload
all: all-recursive
diff --git a/lib/tap-driver b/lib/tap-driver
new file mode 100755
index 0000000..f71536d
--- /dev/null
+++ b/lib/tap-driver
@@ -0,0 +1,279 @@
+#! /usr/bin/env perl
+# Temporary/experimental TAP test driver for Automake.
+# TODO: should be rewritten portably (e.g., in awk or shell).
+
+# ---------------------------------- #
+# Imports, static data, and setup. #
+# ---------------------------------- #
+
+use warnings FATAL => 'all';
+use strict;
+use Getopt::Long ();
+use TAP::Parser;
+
+my $me = "tap-driver";
+
+my $usage = <<'END';
+Usage:
+ tap-driver [--help|--version] --test-name=NAME --log-file=PATH
+ [--expect-failure={yes|no}] [--color-tests={yes|no}]
+ [--enable-hard-errors={yes|no}] [--merge|--no-merge]
+ [--ignore-exit] [--comments|--no-comments] [--] TEST-COMMAND
+The `--test-name' and `--log-file' options are mandatory.
+END
+
+my $help = "$me: TAP-aware test driver for Automake testsuite harness." .
+ "\n" . $usage;
+
+# ----------------- #
+# Option parsing. #
+# ----------------- #
+
+my %cfg = (
+ "color-tests" => 0,
+ "expect-failure" => 0,
+ "enable-hard-errors" => 1,
+ "merge" => 0,
+ "comments" => 0,
+ "ignore-exit" => 0,
+);
+
+my $test_script_name = undef;
+my $log_file = undef;
+
+Getopt::Long::GetOptions (
+ 'help' => sub { print $help; exit 0; },
+ 'test-name=s' => \$test_script_name,
+ 'log-file=s' => \$log_file,
+ 'color-tests=s' => \&bool_opt,
+ 'expect-failure=s' => \&bool_opt,
+ 'enable-hard-errors=s' => \&bool_opt,
+ 'comments' => sub { $cfg{"comments"} = 1; },
+ 'no-comments' => sub { $cfg{"comments"} = 0; },
+ 'merge' => sub { $cfg{"merge"} = 1; },
+ 'no-merge' => sub { $cfg{"merge"} = 0; },
+ 'ignore-exit' => sub { $cfg{"ignore-exit"} = 0; },
+ ) or exit 1;
+
+# --------------------------------------------- #
+# Prototypes, global variables, and main code. #
+# --------------------------------------------- #
+
+sub start (@);
+sub finish ();
+sub main (@);
+
+my $parser;
+
+main @ARGV;
+
+# -------------- #
+# Subroutines. #
+# -------------- #
+
+
+TEST_RESULTS :
+{
+ my (@test_results, %test_results);
+
+ sub add_test_result ($)
+ {
+ my $res = shift;
+ push @test_results, $res;
+ $test_results{$res} = 1;
+ }
+
+ sub get_test_results ()
+ {
+ return @test_results;
+ }
+
+ # FIXME: this can certainly be improved ...
+ sub get_global_test_result ()
+ {
+ my @results = keys %test_results;
+ return "ERROR" if exists $test_results{"ERROR"};
+ return "SKIP" if @results == 1 && $results[0] eq "SKIP";
+ return "FAIL" if exists $test_results{"FAIL"};
+ return "FAIL" if exists $test_results{"XPASS"};
+ return "PASS";
+ }
+
+}
+
+sub bool_opt ($$)
+{
+ my ($opt, $val) = @_;
+ if ($val =~ /^(?:y|yes)\z/i)
+ {
+ $cfg{opt} = 1;
+ }
+ elsif ($val =~ /^(?:n|no)\z/i)
+ {
+ $cfg{opt} = 1;
+ }
+ else
+ {
+ die "invalid argument '$val' for option '$opt'\n";
+ }
+}
+
+sub stringify_test_result ($)
+{
+ my $result = shift;
+ if (!$result->directive)
+ {
+ return $result->is_ok ? "PASS": "FAIL";
+ }
+ elsif ($result->has_todo)
+ {
+ return $result->is_actual_ok ? "XPASS" : "XFAIL";
+ }
+ elsif ($result->has_skip)
+ {
+ return $result->is_ok ? "SKIP" : "FAIL";
+ }
+ die "INTERNAL ERROR"; # NOTREACHED
+}
+
+sub console_output (@)
+{
+ print OLDOUT join ("\n", @_) . "\n" if @_ > 0;
+}
+
+sub all_callback ($)
+{
+ # Verbatim Copy any line of input into the log file.
+ print TMP $_[0]->raw . "\n";
+}
+
+sub test_callback ($)
+{
+ my $test = shift;
+
+ my $test_result = stringify_test_result $test;
+
+ my $string = "$test_result: $test_script_name " . $test->number;
+ if (my $description = $test->description)
+ {
+ $string .= " $description";
+ }
+ if (my $directive = $test->directive)
+ {
+ $string .= " # $directive";
+ if (my $explanation = $test->explanation)
+ {
+ $string .= " $explanation";
+ }
+ }
+
+ add_test_result $test_result;
+ console_output $string;
+}
+
+sub plan_callback ()
+{
+ my $plan = shift;
+ return
+ unless defined $plan->directive && length ($plan->directive) > 0;
+ my $string = "SKIP: $test_script_name";
+ if (my $explanation = $plan->explanation)
+ {
+ $string .= " - $explanation";
+ }
+ add_test_result "SKIP";
+ console_output $string;
+ all_callback ($plan);
+ finish;
+}
+
+sub bailout_callback ($)
+{
+ my $bailout = shift;
+ my $string = "ERROR: $test_script_name - Bail out!";
+ if (my $explanation = $bailout->explanation)
+ {
+ $string .= " $explanation";
+ }
+ add_test_result "ERROR";
+ console_output $string;
+ all_callback ($bailout);
+ finish;
+}
+
+sub comment_callback ($)
+{
+ return unless $cfg{comments};
+ my $comment = $_[0]->comment;
+ console_output "# $test_script_name: $comment"
+ if length $comment;
+}
+
+sub start (@)
+{
+ # Redirect stderr and stdout to a temporary log file. Save the original
+ # stdout stream, since we need it to print testsuite progress output.
+ open OLDOUT, ">&STDOUT" or die "duplicating stdout: $!\n";
+ open TMP, ">$log_file-t" or die "opening $log_file-t: $!\n";
+ open STDOUT, ">&TMP" or die "redirecting stdout: $!\n";
+ open STDERR, ">&TMP" or die "redirecting stderr: $!\n";
+
+ my %callbacks = (
+ test => \&test_callback,
+ plan => \&plan_callback,
+ comment => \&comment_callback,
+ bailout => \&bailout_callback,
+ unknown => sub {},
+ ALL => \&all_callback,
+ );
+
+ $parser = TAP::Parser->new ({
+ exec => address@hidden,
+ callbacks => \%callbacks,
+ merge => $cfg{merge},
+ });
+}
+
+sub finish ()
+{
+ open LOG, ">", $log_file or die "opening $log_file: $!\n";
+
+ # We need to declare a global test result in order to have
+ # "make recheck" working.
+ # FIXME: this should be "SKIP" in case the whole script has
+ # been skipped.
+ my $global_result = get_global_test_result;
+ my $global_result_line = "$global_result: $test_script_name";
+ print LOG "$global_result_line\n";
+ print LOG "=" x length ($global_result_line) . "\n";
+ print LOG "\n";
+
+ foreach (get_test_results)
+ {
+ print LOG ":test-result: $_\n";
+ # Add extra trailing empty lines to support reStructuredText -> HTML
+ # conversion.
+ print LOG "\n";
+ }
+
+ # So that the output from the test script won't be parsed for test
+ # results (which would potentially expose us to false positives).
+ print LOG ":test-result: END\n";
+ print LOG "\n";
+
+ close TMP or die "closing $log_file-t: $!\n";
+ my $test_output = `cat $log_file-t && rm -f $log_file-t`;
+ print LOG $test_output;
+ close LOG or die "closing $log_file: $!\n";
+
+ exit 0;
+}
+
+sub main (@)
+{
+ start @_;
+ $parser->run;
+ finish;
+}
+
+# vim: ft=perl ts=4 sw=4 et
diff --git a/tests/Makefile.am b/tests/Makefile.am
index 0d41a1c..da41369 100644
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -980,6 +980,43 @@ suffix13.test \
symlink.test \
symlink2.test \
syntax.test \
+tap-autonumber.test \
+tap-bad-prog.test \
+tap-bailout.test \
+tap-basic.test \
+tap-color.test \
+tap-deps.test \
+tap-diagnostic.test \
+tap-empty-diagnostic.test \
+tap-escape-directive.test \
+tap-exit.test \
+tap-fancy.test \
+tap-fancy2.test \
+tap-global-result.test \
+tap-html.test \
+tap-log.test \
+tap-merge-stdout-stderr.test \
+tap-more.test \
+tap-more2.test \
+tap-no-merge-stdout-stderr.test \
+tap-no-spurious-summary.test \
+tap-no-spurious.test \
+tap-not-ok-skip.test \
+tap-numeric-description.test \
+tap-out-of-order.test \
+tap-passthrough.test \
+tap-plan-errors.test \
+tap-plan.test \
+tap-realtime.test \
+tap-recheck-logs.test \
+tap-recheck.test \
+tap-skip-whole.test \
+tap-summary.test \
+tap-todo-skip-together.test \
+tap-todo-skip.test \
+tap-whitespace-normalization.test \
+tap-with-and-without-number.test \
+tap-xfail-tests.test \
tags.test \
tags2.test \
tagsub.test \
diff --git a/tests/Makefile.in b/tests/Makefile.in
index c72b1cf..1ad2ca0 100644
--- a/tests/Makefile.in
+++ b/tests/Makefile.in
@@ -1227,6 +1227,43 @@ suffix13.test \
symlink.test \
symlink2.test \
syntax.test \
+tap-autonumber.test \
+tap-bad-prog.test \
+tap-bailout.test \
+tap-basic.test \
+tap-color.test \
+tap-deps.test \
+tap-diagnostic.test \
+tap-empty-diagnostic.test \
+tap-escape-directive.test \
+tap-exit.test \
+tap-fancy.test \
+tap-fancy2.test \
+tap-global-result.test \
+tap-html.test \
+tap-log.test \
+tap-merge-stdout-stderr.test \
+tap-more.test \
+tap-more2.test \
+tap-no-merge-stdout-stderr.test \
+tap-no-spurious-summary.test \
+tap-no-spurious.test \
+tap-not-ok-skip.test \
+tap-numeric-description.test \
+tap-out-of-order.test \
+tap-passthrough.test \
+tap-plan-errors.test \
+tap-plan.test \
+tap-realtime.test \
+tap-recheck-logs.test \
+tap-recheck.test \
+tap-skip-whole.test \
+tap-summary.test \
+tap-todo-skip-together.test \
+tap-todo-skip.test \
+tap-whitespace-normalization.test \
+tap-with-and-without-number.test \
+tap-xfail-tests.test \
tags.test \
tags2.test \
tagsub.test \
diff --git a/tests/tap-autonumber.test b/tests/tap-autonumber.test
new file mode 100755
index 0000000..7a72aec
--- /dev/null
+++ b/tests/tap-autonumber.test
@@ -0,0 +1,94 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support:
+# - unnumbered tests are OK, as long as their final total number
+# agrees with the plan
+# - test results without number get automatically numbered in the
+# console progress output
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TEST_LOG_COMPILER = cat
+TESTS = all.test
+EXTRA_DIST = $(TESTS)
+END
+
+cat > all.test <<'END'
+1..16
+ok
+ok foo
+ok - foo2
+not ok
+not ok bar
+not ok - bar2
+; See that we can intermingle different kind of results without
+; messing up the autonumbering
+ok
+ok # TODO
+not ok # TODO who cares?
+ok
+not ok
+ok muuu # SKIP
+not ok
+ok
+END
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+$MAKE check >stdout && { cat stdout; Exit 1; }
+cat stdout
+count_test_results total=14 pass=6 fail=5 xpass=1 xfail=1 skip=1 error=0
+
+cat > exp <<'END'
+PASS: all.test 1
+PASS: all.test 2 foo
+PASS: all.test 3 - foo2
+FAIL: all.test 4
+FAIL: all.test 5 bar
+FAIL: all.test 6 - bar2
+PASS: all.test 7
+XPASS: all.test 8 # TODO
+XFAIL: all.test 9 # TODO who cares?
+PASS: all.test 10
+FAIL: all.test 11
+SKIP: all.test 12 muuu # SKIP
+FAIL: all.test 13
+PASS: all.test 14
+END
+
+$FGREP ': all.test' stdout > got
+
+cat exp
+cat got
+diff exp got
+
+:
diff --git a/tests/tap-bad-prog.test b/tests/tap-bad-prog.test
new file mode 100755
index 0000000..af54b2e
--- /dev/null
+++ b/tests/tap-bad-prog.test
@@ -0,0 +1,69 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support:
+# - missing, unreadable, or not-executable test scripts cause proper
+# error reports
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TESTS = none.test noread.test noexec.test
+none.test:
+END
+
+cat > noexec.test <<'END'
+#!/bin/sh
+echo 1..1
+ok 1
+END
+
+./noexec.test && skip_ "can't have non-executable files"
+
+cp noexec.test noread.test
+chmod a-r noread
+
+if test -r noread.test || cat noread.test; then
+ skip_ "can't have readonly files"
+fi
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+$MAKE check >stdout && { cat stdout; Exit 1; }
+cat stdout
+
+# FIXME: maybe better error messages?
+grep '^ERROR: none.test' stdout
+grep '^ERROR: noexec.test' stdout
+grep '^ERROR: noread.test' stdout
+
+count_test_results total=3 pass=0 fail=0 xpass=0 xfail=0 skip=0 error=3
+
+:
diff --git a/tests/tap-bailout.test b/tests/tap-bailout.test
new file mode 100755
index 0000000..6c0a588
--- /dev/null
+++ b/tests/tap-bailout.test
@@ -0,0 +1,155 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# Basic TAP test protocol support:
+# - "Bail out!" magic
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TEST_LOG_COMPILER = cat
+TESTS = a.test b.test c.test d.test e.test
+END
+
+: > exp
+
+#------------------------------------------------------------------
+
+# Bailout without explanation.
+
+cat > a.test <<END
+1..4
+ok 1
+not ok 2
+Bail out!
+not ok 3
+ok 4 # SKIP
+END
+
+cat >> exp <<END
+PASS: a.test 1
+FAIL: a.test 2
+ERROR: a.test - Bail out!
+END
+
+# pass += 1, fail +=1, error += 1
+
+#------------------------------------------------------------------
+
+# Bailout with explanation.
+
+cat > b.test <<END
+1..7
+ok 1 # SKIP
+ok 2 # TODO
+not ok 3 # TODO
+Bail out! We're out of disk space.
+ok 4
+not ok 5
+not ok 6 # TODO
+ok 7 # TODO
+END
+
+cat >> exp <<END
+SKIP: b.test 1 # SKIP
+XPASS: b.test 2 # TODO
+XFAIL: b.test 3 # TODO
+ERROR: b.test - Bail out! We're out of disk space.
+END
+
+# skip += 1, xpass += 1, xfail += 1, error += 1
+
+#------------------------------------------------------------------
+
+# Bail out before the test plan.
+
+cat > c.test <<END
+ok 1
+ok 2
+Bail out! BOOOH!
+1..2
+END
+
+cat >> exp <<END
+PASS: c.test 1
+PASS: c.test 2
+ERROR: c.test - Bail out! BOOOH!
+END
+
+# pass += 2, error += 1
+
+#------------------------------------------------------------------
+
+# Bailout on the first line.
+
+cat > d.test <<END
+Bail out! mktemp -d: Permission denied
+ok 1
+END
+
+echo 'ERROR: d.test - Bail out! mktemp -d: Permission denied' >> exp
+
+# error += 1
+
+#------------------------------------------------------------------
+
+# TAP input comprised only of a bailout directive.
+
+cat > e.test <<END
+Bail out!
+END
+
+echo "ERROR: e.test - Bail out!" >> exp
+
+# error += 1
+
+#------------------------------------------------------------------
+
+# Doing the sums above, we have:
+test_counts='total=12 pass=3 fail=1 xpass=1 xfail=1 skip=1 error=5'
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+$MAKE check >stdout && { cat stdout; Exit 1; }
+cat stdout
+
+count_test_results $test_counts
+
+LC_ALL=C sort exp > t
+mv -f t exp
+
+# We need the sort below to account for parallel make usage.
+grep ': [abcde]\.test' stdout | LC_ALL=C sort > got
+
+cat exp
+cat got
+diff exp got
+
+:
diff --git a/tests/tap-basic.test b/tests/tap-basic.test
new file mode 100755
index 0000000..26a749d
--- /dev/null
+++ b/tests/tap-basic.test
@@ -0,0 +1,177 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# Basic TAP support:
+# - LOG_COMPILER support;
+# - basic support for TODO and SKIP directives, and "Bail out!" magic;
+# - testsuite progress output on console;
+# - runtime overriding of TESTS and TEST_LOGS;
+# - correct counts of test results (both in summary and in progress
+# output on console).
+# Note that some of the features checked here are checked in other
+# test cases too, usually in a more thorough and detailed way.
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+## Defining LOG_COMPILER should work and not intefere with the
+## tap-driver script.
+TEST_LOG_COMPILER = cat
+TESTS = success.test
+
+ok.test:
+ echo '1..3' > address@hidden
+ echo 'ok 1' >> address@hidden
+ echo 'not ok 2 # TODO' >>address@hidden
+ echo 'ok 3 # SKIP' >>address@hidden
+ cat address@hidden ;: For debugging.
+ mv -f address@hidden $@
+END
+
+cat > success.test << 'END'
+1..20
+ok 1
+ok 2 two
+ok 3 - three
+ok 4 four four
+not ok 5
+not ok 6 six
+not ok 7 - seven
+not ok 8 eight eight
+ok 9 # TODO
+ok 10 ten # TODO
+ok 11 - eleven # TODO
+ok 12 twelve twelve # TODO
+not ok 13 # TODO
+not ok 14 fourteen # TODO
+not ok 15 - fifteen # TODO
+not ok 16 sixteen sixteen # TODO
+ok 17 # SKIP
+ok 18 eighteen # SKIP
+ok 19 - nineteen # SKIP
+ok 20 twenty twenty # SKIP
+END
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+# Basilar usage and testsuite progress output.
+
+$MAKE check >stdout && { cat stdout; Exit 1; }
+cat stdout
+
+count_test_results total=20 pass=4 fail=4 xpass=4 xfail=4 skip=4 error=0
+
+test -f success.log
+test -f test-suite.log
+
+cat > exp << 'END'
+PASS: success.test 1
+PASS: success.test 2 two
+PASS: success.test 3 - three
+PASS: success.test 4 four four
+FAIL: success.test 5
+FAIL: success.test 6 six
+FAIL: success.test 7 - seven
+FAIL: success.test 8 eight eight
+XPASS: success.test 9 # TODO
+XPASS: success.test 10 ten # TODO
+XPASS: success.test 11 - eleven # TODO
+XPASS: success.test 12 twelve twelve # TODO
+XFAIL: success.test 13 # TODO
+XFAIL: success.test 14 fourteen # TODO
+XFAIL: success.test 15 - fifteen # TODO
+XFAIL: success.test 16 sixteen sixteen # TODO
+SKIP: success.test 17 # SKIP
+SKIP: success.test 18 eighteen # SKIP
+SKIP: success.test 19 - nineteen # SKIP
+SKIP: success.test 20 twenty twenty # SKIP
+END
+
+$FGREP ': success.test' stdout > got
+
+cat exp
+cat got
+diff exp got
+
+# Override TESTS from the command line.
+
+rm -f *.log *.test
+
+cat > bail.test <<'END'
+1..1
+Bail out!
+ok 1
+END
+
+TESTS=bail.test $MAKE -e check >stdout && { cat stdout; Exit 1; }
+cat stdout
+
+count_test_results total=1 pass=0 fail=0 xpass=0 xfail=0 skip=0 error=1
+
+test ! -f success.log
+test -f bail.log
+test -f test-suite.log
+
+grep '^ERROR: bail\.test - Bail out!' stdout
+grep '^PASS:' stdout && Exit 1
+test `$FGREP -c ': bail.test' stdout` -eq 1
+$FGREP 'success.test' stdout && Exit 1
+
+# Override TEST_LOGS from the command line, making it point to a test
+# (ok.test) that have to be generated at make time.
+
+rm -f *.log *.test
+
+TEST_LOGS=ok.log $MAKE -e check >stdout || { cat stdout; Exit 1; }
+cat stdout
+
+count_test_results total=3 pass=1 fail=0 xpass=0 xfail=1 skip=1 error=0
+
+test -f ok.test
+test -f ok.log
+test ! -f success.log
+test ! -f bail.log
+test -f test-suite.log
+
+$EGREP '(bail|success)\.test' stdout && Exit 1
+
+cat > exp << 'END'
+PASS: ok.test 1
+XFAIL: ok.test 2 # TODO
+SKIP: ok.test 3 # SKIP
+END
+
+$FGREP ': ok.test' stdout > got
+
+cat exp
+cat got
+diff exp got
+
+:
diff --git a/tests/tap-color.test b/tests/tap-color.test
new file mode 100755
index 0000000..8ed7265
--- /dev/null
+++ b/tests/tap-color.test
@@ -0,0 +1,145 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support:
+# - colorization of TAP results and diagnostic messages
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+TERM=ansi; export TERM
+
+esc=''
+# Escape `[' for grep, below.
+red="$esc\[0;31m"
+grn="$esc\[0;32m"
+lgn="$esc\[1;32m"
+blu="$esc\[1;34m"
+mgn="$esc\[0;35m"
+std="$esc\[m"
+
+# Check that grep can parse nonprinting characters.
+# BSD 'grep' works from a pipe, but not a seekable file.
+# GNU or BSD 'grep -a' works on files, but is not portable.
+case `echo "$std" | grep .` in
+ "$std") ;;
+ *) echo "$me: grep can't parse nonprinting characters" >&2; Exit 77;;
+esac
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+AUTOMAKE_OPTIONS = color-tests
+AM_TEST_LOG_DRIVER_FLAGS = --comments
+TEST_LOG_COMPILER = cat
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TESTS = all.test skip.test bailout.test plan-errors.test
+END
+
+cat > all.test << 'END'
+1..5
+ok 1 - foo
+# Hi! I shouldn't be colorized!
+not ok 2 - bar # TODO td
+ok 3 - baz # SKIP sk
+not ok 4 - quux
+ok 5 - zardoz # TODO
+END
+
+cat > skip.test << 'END'
+1..0 # Whole script not run
+END
+
+cat > errors.test << 'END'
+1..1
+ok 1
+Bail out!
+END
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+AM_COLOR_TESTS=always $MAKE check >stdout && { cat stdout; Exit 1; }
+cat stdout
+
+test_color ()
+{
+ # Not a useless use of cat; see above comments about grep.
+ cat stdout | grep "^${grn}PASS${std}: all\.test 1 - foo$"
+ cat stdout | grep "^${lgn}XFAIL${std}: all\.test 2 - bar # TODO td$"
+ cat stdout | grep "^${blu}SKIP${std}: all\.test 3 - baz # SKIP sk$"
+ cat stdout | grep "^${red}FAIL${std}: all\.test 4 - quux$"
+ cat stdout | grep "^${red}XPASS${std}: all\.test 5 - zardoz # TODO$"
+ cat stdout | grep "^${blu}SKIP${std}: skip\.test # Whole script not run$"
+ cat stdout | grep "^${grn}PASS${std}: bailout\.test 1$"
+ cat stdout | grep "^${mgn}ERROR${std}: all\.test - Bail out!$"
+ # Diagnostic messages shouldn't be colorized.
+ cat stdout | grep "^# all\.test: Hi! I shouldn't be colorized!$"
+ :
+}
+
+test_no_color ()
+{
+ # With make implementations that, like Solaris make, in case of errors
+ # print the whole failing recipe on standard output, we should content
+ # ourselves with a laxer check, to avoid false positives.
+ # Keep this in sync with lib/am/check.am:$(am__color_tests).
+ if $FGREP '= Xalways || test -t 1 ' stdout; then
+ # Extra verbose make, resort to laxer checks.
+ # But we also want to check that the testsuite summary is not unduly
+ # colorized.
+ (
+ set +e # In case some grepped regex below isn't matched.
+ # Not a useless use of cat; see above comments about grep.
+ cat stdout | grep "TOTAL.*:"
+ cat stdout | grep "PASS.*:"
+ cat stdout | grep "FAIL.*:"
+ cat stdout | grep "SKIP.*:"
+ cat stdout | grep "XFAIL.*:"
+ cat stdout | grep "XPASS.*:"
+ cat stdout | grep "ERROR.*:"
+ cat stdout | grep "^#"
+ cat stdout | grep 'test.*expected'
+ cat stdout | grep 'test.*not run'
+ cat stdout | grep '===='
+ cat stdout | grep '[Ss]ee .*test-suite\.log'
+ cat stdout | grep '[Tt]estsuite summary'
+ ) | grep "$esc" && Exit 1
+ : For shells with broken 'set -e'
+ else
+ cat stdout | grep "$esc" && Exit 1
+ : For shells with broken 'set -e'
+ fi
+}
+
+AM_COLOR_TESTS=always $MAKE check >stdout && { cat stdout; Exit 1; }
+cat stdout
+test_color
+
+$MAKE -e check >stdout && { cat stdout; Exit 1; }
+cat stdout
+test_no_color
+
+:
diff --git a/tests/tap-deps.test b/tests/tap-deps.test
new file mode 100755
index 0000000..ec74485
--- /dev/null
+++ b/tests/tap-deps.test
@@ -0,0 +1,98 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# Basic TAP test protocol support:
+# - dependencies between test scripts
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+# The tests are *deliberately* listed in inversed order here.
+TESTS = c.test b.test a.test
+b.log: a.log
+c.log: b.log
+END
+
+cat > a.test << 'END'
+#!/bin/sh
+echo 1..2
+echo ok 1
+sleep '3' # FIXME: quotes to please maintainer-check
+echo ok 2
+: > a.run
+END
+
+cat > b.test << 'END'
+#!/bin/sh
+echo 1..2
+if test -f a.run; then
+ echo ok 1
+else
+ echo not ok 1
+fi
+sleep '3' # FIXME: quotes to please maintainer-check
+echo ok 2
+: > b.run
+END
+
+cat > c.test << 'END'
+#!/bin/sh
+echo 1..1
+test -f b.run || { echo 'Bail out!'; exit 1; }
+echo ok 1
+rm -f a.run b.run
+END
+
+chmod a+x *.test
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+$MAKE check >stdout || { cat stdout; Exit 1; }
+cat stdout
+
+count_test_results total=5 pass=5 fail=0 xpass=0 xfail=0 skip=0 error=0
+
+cat > exp << 'END'
+PASS: a.test 1
+PASS: a.test 2
+PASS: b.test 1
+PASS: b.test 2
+PASS: c.test 1
+END
+
+grep ': [abc]\.test' stdout > got
+
+cat exp
+cat got
+diff exp got
+
+# TODO: it would be nice to also redo the checks forcing parallel make...
+
+:
diff --git a/tests/tap-diagnostic.test b/tests/tap-diagnostic.test
new file mode 100755
index 0000000..b70b88c
--- /dev/null
+++ b/tests/tap-diagnostic.test
@@ -0,0 +1,142 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support:
+# - diagnostic messages (TAP lines with leading "#")
+# - flags '--comments' and '--no-comments' of the TAP test driver
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TEST_LOG_COMPILER = cat
+TESTS = all.test
+END
+
+metacharacters=\''"\$!&()[]<>#;^?*'
+
+cat > all.test <<END
+1..4
+# Hi! I'm a comment.
+# Tests begin.
+ok 1
+not ok 2 - foo # TODO
+ok 3 - bar # SKIP
+# Tests end.
+ok - zardoz
+# Shell metacharacters here: $metacharacters
+.# Leading characters before "#", not a TAP diagnostic line.
+x # Leading characters before "#", not a TAP diagnostic line.
+ # Leading whitespace before "#", not a TAP diagnostic line.
+${tab}# Leading whitespace before "#", not a TAP diagnostic line.
+ ${tab} # Leading whitespace before "#", not a TAP diagnostic line.
+END
+
+cat > exp <<END
+# all.test: Hi! I'm a comment.
+# all.test: Tests begin.
+PASS: all.test 1
+XFAIL: all.test 2 - foo # TODO
+SKIP: all.test 3 - bar # SKIP
+# all.test: Tests end.
+PASS: all.test 4 - zardoz
+# all.test: Shell metacharacters here: $metacharacters
+END
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+$MAKE check >stdout || { cat stdout; Exit 1; }
+cat stdout
+grep '#.*all\.test' stdout && Exit 1
+count_test_results total=4 pass=2 fail=0 xpass=0 xfail=1 skip=1 error=0
+
+echo 'AM_TEST_LOG_DRIVER_FLAGS = --comments' >> Makefile
+$MAKE check >stdout || { cat stdout; Exit 1; }
+cat stdout
+$FGREP ' all.test' stdout > got
+cat exp
+cat got
+diff exp got
+count_test_results total=4 pass=2 fail=0 xpass=0 xfail=1 skip=1 error=0
+
+TEST_LOG_DRIVER_FLAGS="--no-comments" $MAKE -e check >stdout \
+ || { cat stdout; Exit 1; }
+cat stdout
+grep '#.*all\.test' stdout && Exit 1
+count_test_results total=4 pass=2 fail=0 xpass=0 xfail=1 skip=1 error=0
+
+# The "#"-prepended lines here shouldn't be parsed as test results.
+cat > all.test <<END
+1..1
+ok
+# ok
+#ok
+# not ok
+#not ok
+# Bail out!
+#Bail out!
+# SKIP
+#SKIP
+# TODO
+#TODO
+END
+
+$MAKE check >stdout || { cat stdout; Exit 1; }
+cat stdout
+count_test_results total=1 pass=1 fail=0 xpass=0 xfail=0 skip=0 error=0
+
+# Diagnostic wihout leading whitespace, or with extra leading whitespace,
+# is ok. Be laxer in the grepping checks, to allow for whitespace
+# normalization.
+
+ws="[ $tab]"
+ws0p="${ws}*"
+ws1p="${ws}${ws0p}"
+
+cat > all.test <<END
+1..1
+ok 1
+#foo
+#bar${tab}
+# zardoz ${tab}
+# ${tab} ${tab}${tab}foo bar${tab}baz ${tab}
+END
+
+$MAKE check >stdout || { cat stdout; Exit 1; }
+cat stdout
+count_test_results total=1 pass=1 fail=0 xpass=0 xfail=0 skip=0 error=0
+
+grep "^# all.test:${ws0p}foo$" stdout
+grep "^# all.test:${ws0p}bar${ws0p}$" stdout
+grep "^# all.test:${ws1p}zardoz${ws0p}$" stdout
+grep "^# all.test:${ws1p}foo${ws1p}bar${ws1p}baz${ws0p}$" stdout
+
+test `grep -c '^# all\.test:' stdout` -eq 4
+
+:
diff --git a/tests/tap-empty-diagnostic.test b/tests/tap-empty-diagnostic.test
new file mode 100755
index 0000000..0edea13
--- /dev/null
+++ b/tests/tap-empty-diagnostic.test
@@ -0,0 +1,59 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support:
+# - empty diagnostic messages are discarder
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TEST_LOG_COMPILER = cat
+AM_TEST_LOG_DRIVER_FLAGS = --comments
+TESTS = all.test
+END
+
+sed 's/\$$//' > all.test <<END
+1..1$
+ok 1$
+#$
+# $
+#${tab}$
+# ${tab} $tab${tab}$
+END
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+$MAKE check >stdout || { cat stdout; Exit 1; }
+cat stdout
+
+grep '#.*all\.test' stdout && Exit 1
+grep "all\.test[ $tab]*:[ $tab]*$" stdout && Exit 1
+
+:
diff --git a/tests/tap-escape-directive.test b/tests/tap-escape-directive.test
new file mode 100755
index 0000000..fb333ba
--- /dev/null
+++ b/tests/tap-escape-directive.test
@@ -0,0 +1,57 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support:
+# - "escape" TODO and SKIP directives (by escaping the "#" character)
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TEST_LOG_COMPILER = cat
+TESTS = all.test
+END
+
+cat > all.test <<'END'
+1..2
+ok \# TODO
+ok \# SKIP
+END
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+$MAKE check >stdout || { cat stdout; Exit 1; }
+cat stdout
+
+count_test_results total=2 pass=2 fail=0 xpass=0 xfail=0 skip=0 error=0
+
+grep '^PASS: all\.test 1 .*TODO' stdout
+grep '^PASS: all\.test 2 .*SKIP' stdout
+
+:
diff --git a/tests/tap-exit.test b/tests/tap-exit.test
new file mode 100755
index 0000000..be464a5
--- /dev/null
+++ b/tests/tap-exit.test
@@ -0,0 +1,76 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support:
+# - an exit status != 0 of a test script causes an hard error
+# - the `--ignore-exit' option causes the TAP test driver to ignore
+# exit statuses of the test scripts.
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TESTS = signal.test
+END
+
+cat > signal.test <<'END'
+#! /bin/sh
+echo 1..1
+echo ok 1
+kill $$
+END
+
+for st in 1 2 77 99; do
+ unindent > exit$i.test <<END
+ #!/bin/sh
+ echo 1..1
+ echo ok 1
+ exit $st
+END
+ echo TESTS += exit$i.test >> Makefile.am
+done
+
+chmod a+x *.test
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+$MAKE check >stdout && { cat stdout; Exit 1; }
+cat stdout
+count_test_results total=10 pass=5 fail=0 xpass=0 xfail=0 skip=0 error=5
+
+grep '^ERROR: exit1\.test - exited with status 1$' stdout
+grep '^ERROR: exit2\.test - exited with status 2$' stdout
+grep '^ERROR: exit77\.test - exited with status 77$' stdout
+grep '^ERROR: exit99\.test - exited with status 99$' stdout
+grep '^ERROR: signal\.test - terminated by signal.*' stdout
+
+echo TEST_LOG_DRIVER_FLAGS = --ignore-exit >> Makefile
+$MAKE check >stdout && { cat stdout; Exit 1; }
+cat stdout
+count_test_results total=5 pass=5 fail=0 xpass=0 xfail=0 skip=0 error=0
+
+:
diff --git a/tests/tap-fancy.test b/tests/tap-fancy.test
new file mode 100755
index 0000000..68612de
--- /dev/null
+++ b/tests/tap-fancy.test
@@ -0,0 +1,151 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support: some unusual forms for valid TAP input.
+# See also related test 'tap-fancy2.test'.
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TEST_LOG_COMPILER = cat
+TESTS = all.test
+END
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+#
+# From manpage Test::Harness::TAP(3):
+#
+# ``Lines written to standard output matching /^(not )?ok\b/ must be
+# interpreted as test lines. All other lines must not be considered
+# test output.''
+#
+# Unfortunately, the exact format of TODO and SKIP directives is not as
+# clearly described in that manpage; but a simple reverse-engineering of
+# the prove(1) utility shows that it is probably given by the perl regex
+# /#\s*(TODO|SKIP)\b/.
+#
+
+cat > all.test <<END
+1..21
+
+ok? a question
+not ok? a question
+
+ok+plus
+not ok+plus
+
+ok-minus
+not ok-minus
+
+ok#55
+not ok#55
+
+ok${tab} ${tab}9
+ok ${tab}${tab} 10
+
+not ok${tab} ${tab}11
+not ok ${tab}${tab} 12
+
+ok# SKIP
+ok${tab}#SKIP--who cares?
+ok?#SKIP!
+ok!#SKIP?
+
+not ok# TODO
+not ok${tab}#TODO--who cares?
+not ok?#TODO!
+not ok!#TODO?
+
+ok~#TODO
+END
+
+$MAKE check >stdout && { cat stdout; Exit 1; }
+cat stdout
+
+count_test_results total=21 pass=6 fail=6 xfail=4 xpass=1 skip=4 error=0
+
+#
+# "Weird" characters support.
+#
+
+# The "#" character might cause confusion w.r.t. TAP directives (TODO,
+# SKIP), so we don't attempt to use it.
+weirdchars=\''"$!&()[]<>;^?*/@%=,.:'
+
+cat > all.test <<END
+1..6
+ok $weirdchars
+not ok $weirdchars
+ok $weirdchars # TODO
+not ok $weirdchars # TODO
+ok $weirdchars # SKIP
+Bail out! $weirdchars
+END
+
+$MAKE check >stdout && { cat stdout; Exit 1; }
+cat stdout
+
+count_test_results total=6 pass=1 fail=1 xfail=1 xpass=1 skip=1 error=1
+$FGREP "PASS: all.test 1 $weirdchars" stdout
+$FGREP "FAIL: all.test 2 $weirdchars" stdout
+$FGREP "XPASS: all.test 3 $weirdchars" stdout
+$FGREP "XFAIL: all.test 4 $weirdchars" stdout
+$FGREP "SKIP: all.test 5 $weirdchars" stdout
+$FGREP "ERROR: all.test - Bail out! $weirdchars" stdout
+
+#
+# Trailing backslashes does not confuse the parser.
+#
+
+bs='\'
+
+cat > all.test <<END
+1..6
+ok $bs
+not ok $bs
+ok # TODO $bs
+not ok # TODO $bs
+ok # SKIP $bs
+Bail out! $bs
+END
+
+$MAKE check >stdout && { cat stdout; Exit 1; }
+cat stdout
+
+count_test_results total=6 pass=1 fail=1 xfail=1 xpass=1 skip=1 error=1
+grep '^PASS: all\.test 1 \\$' stdout
+grep '^FAIL: all\.test 2 \\$' stdout
+grep '^XPASS: all\.test 3 # TODO \\$' stdout
+grep '^XFAIL: all\.test 4 # TODO \\$' stdout
+grep '^SKIP: all\.test 5 # SKIP \\$' stdout
+grep '^ERROR: all\.test - Bail out! \\$' stdout
+
+:
diff --git a/tests/tap-fancy2.test b/tests/tap-fancy2.test
new file mode 100755
index 0000000..b97d787
--- /dev/null
+++ b/tests/tap-fancy2.test
@@ -0,0 +1,147 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support: more unusual forms for valid TAP input.
+# See also related test 'tap-fancy.test'.
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TEST_LOG_COMPILER = cat
+TESTS = all.test
+END
+
+#
+# From manpage Test::Harness::TAP(3):
+#
+# ``Lines written to standard output matching /^(not )?ok\b/ must be
+# interpreted as test lines. All other lines must not be considered
+# test output.''
+#
+# Unfortunately, the exact format of TODO and SKIP directives is not as
+# clearly described in that manpage; but a simple reverse-engineering of
+# the prove(1) utility shows that it is probably given by the perl regex
+# /#\s*(TODO|SKIP)\b/.
+#
+
+# There are 34 values for $str ...
+for str in \
+ \' \
+ '"' \
+ '`' \
+ '#' \
+ '$' \
+ '!' \
+ '\' \
+ '/' \
+ '&' \
+ '%' \
+ '(' \
+ ')' \
+ '|' \
+ '^' \
+ '~' \
+ '?' \
+ '*' \
+ '+' \
+ '-' \
+ ',' \
+ ':' \
+ ';' \
+ '=' \
+ '<' \
+ '>' \
+ '@' \
+ '[' \
+ ']' \
+ '{' \
+ '}' \
+ '\\' \
+ '...' \
+ '?[a-zA-Z0-9]*' \
+ '*.*' \
+; do
+ # ... each of them add 1 pass, 1 fail, ...
+ echo "ok${str}"
+ echo "not ok${str}"
+ # ... and (generally) 4 skips, 4 xfails, and 4 xpasses ...
+ for settings in \
+ 'result="ok" directive=SKIP' \
+ 'result="not ok" directive=TODO' \
+ 'result="ok" directive=TODO' \
+ ; do
+ eval "$settings"
+ echo "${result}# ${directive}${str}"
+ # ... but 6 skips, 6 xpasses and 6 xfails are to be removed, since
+ # they might not work with $str = '#' or $str = '\' ...
+ if test x"$str" != x'#' && test x"$str" != x'\'; then
+ echo "${result}${str}#${directive}"
+ echo "${result}${str}# ${tab}${tab} ${directive}"
+ echo "${result}${str}#${directive}${str}"
+ fi
+ done
+done > all.test
+
+# ... so that we finally have:
+pass=34
+fail=34
+xfail=130 # = 4 * 34 - 6
+xpass=130 # = 4 * 34 - 6
+skip=130 # = 4 * 34 - 6
+error=0
+total=`expr $pass + $fail + $xfail + $xpass + $skip`
+
+# Even nastier! But accordingly to the specifics, it should still work.
+for result in 'ok' 'not ok'; do
+ echo "${result}{[(<#${tab}TODO>)]}" >> all.test
+done
+echo "ok{[(<#${tab}SKIP>)]}" >> all.test
+
+# We have to update some test counts.
+xfail=`expr $xfail + 1`
+xpass=`expr $xpass + 1`
+skip=`expr $skip + 1`
+total=`expr $total + 3`
+
+# And add the test plan!
+echo 1..$total >> all.test
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+$MAKE check >stdout && { cat stdout; Exit 1; }
+cat stdout
+
+$EGREP '^(PASS|FAIL|SKIP).*#.*TODO' stdout && Exit 1
+$EGREP '^X?(PASS|FAIL).*#.*SKIP' stdout && Exit 1
+
+count_test_results total=$total pass=$pass fail=$fail skip=$skip \
+ xpass=$xpass xfail=$xfail error=$error
+
+:
diff --git a/tests/tap-global-result.test b/tests/tap-global-result.test
new file mode 100755
index 0000000..d2e593e
--- /dev/null
+++ b/tests/tap-global-result.test
@@ -0,0 +1,128 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support:
+# - which global test result derives from different test results
+# mixed in a single script?
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TEST_LOG_COMPILER = cat
+TESTS = \
+ ok.test \
+ skip.test skipall.test \
+ fail.test fail2.test \
+ xpass.test xpass2.test \
+ error.test error2.test \
+ hodgepodge.test hodgepodge-all.test
+END
+
+cat > ok.test <<END
+1..3
+ok 1
+not ok 2 # TODO
+ok 3 # SKIP
+END
+
+cat > skip.test <<'END'
+1..3
+ok 1 # SKIP
+ok 2 # SKIP
+ok 3 # SKIP
+END
+
+cat > skipall.test <<'END'
+1..0 # SKIP
+not ok 1
+END
+
+cat > fail.test <<'END'
+1..1
+not ok 1
+END
+
+(sed '1s/.*/1..4/' ok.test && echo 'not ok 4') > fail2.test
+
+cat > xpass.test <<'END'
+1..1
+ok 1 # TODO
+END
+
+(sed '1s/.*/1..4/' ok.test && echo 'ok 4 # TODO') > xpass2.test
+
+echo 'Bail out!' > error.test
+(cat ok.test && echo 'Bail out!') > error2.test
+
+cat > hodgepodge.test <<'END'
+1..2
+not ok 1
+ok 2 # TODO
+Bail out!
+END
+
+cat > hodgepodge-all.test <<'END'
+1..4
+ok 1
+ok 2 # SKIP
+not ok 2 # TODO
+not ok 3
+ok 4 # TODO
+Bail out!
+END
+
+# TODO: add scripts with TAP errors (multiple plans, out-of-order
+# tests, etc).
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+$MAKE check >stdout && { cat stdout; Exit 1; }
+cat stdout
+
+for tst in ok skip skipall fail fail2 xpass xpass2 error error2 \
+ hodgepodge hodgepodge-all; do
+ sed -e 2q $tst.log > $tst.res
+done
+
+cat *.res # For debugging.
+
+grep '^PASS:' ok.res
+grep '^SKIP:' skip.res
+grep '^SKIP:' skipall.res
+grep '^FAIL:' fail.res
+grep '^FAIL:' fail2.res
+grep '^FAIL:' xpass.res
+grep '^FAIL:' xpass2.res
+grep '^ERROR:' error.res
+grep '^ERROR:' error2.res
+grep '^ERROR:' hodgepodge.res
+grep '^ERROR:' hodgepodge-all.res
+
+:
diff --git a/tests/tap-html.test b/tests/tap-html.test
new file mode 100755
index 0000000..12dcb36
--- /dev/null
+++ b/tests/tap-html.test
@@ -0,0 +1,93 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support:
+# - "check-html" and "recheck-html" targets
+# - reStructuredText -> HTML conversion
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TEST_LOG_COMPILER = cat
+TESTS = ok.test ko.test
+END
+
+cat > ok.test << 'END'
+1..3
+ok 1 - expected success
+not ok 2 - expectred failure # TODO
+ok 3 - skipped test # SKIP
+END
+
+cat > ko.test << 'END'
+1..2
+not ok 1 - unexpected failure
+ok 2 - spurious success # TODO
+Bail out! hard error
+END
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+no_rst2html_error ()
+{
+ $EGREP '(test-suite\.log:[0-9]|WARNING|SEVERE)' output && Exit 1
+ :
+}
+
+# Overriding TESTS should work with check-html.
+TESTS=ok.test $MAKE -e check-html >output 2>&1 || { cat output; Exit 1; }
+cat output
+no_rst2html_error
+test -f test-suite.html
+
+mv output stdout
+count_test_results total=3 pass=1 fail=0 xpass=0 xfail=1 skip=1 error=0
+
+: > older
+$sleep
+$MAKE check-html >output 2>&1 && { cat output; Exit 1; }
+cat output
+no_rst2html_error
+# test-suite.html should be remake if if it was up-to-date.
+test -f test-suite.html
+is_newest test-suite.html older
+
+mv output stdout
+count_test_results total=6 pass=1 fail=1 xpass=1 xfail=1 skip=1 error=1
+
+rm -f ok.test
+$MAKE recheck-html >output 2>&1 && { cat output; Exit 1; }
+cat output
+no_rst2html_error
+
+mv output stdout
+count_test_results total=3 pass=0 fail=1 xpass=1 xfail=0 skip=0 error=1
+
+:
diff --git a/tests/tap-log.test b/tests/tap-log.test
new file mode 100755
index 0000000..37b5956
--- /dev/null
+++ b/tests/tap-log.test
@@ -0,0 +1,174 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support:
+# - log file creation
+# - log file removal
+# - stdout and stderr of a TAP-generating test script go in its log file
+# - TEST_SUITE_LOG redefinition, at either automake or make time
+# - VERBOSE environment variable support
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TESTS = pass.test skip.test xfail.test fail.test xpass.test error.test
+TEST_SUITE_LOG = global.log
+END
+
+# Custom markers, for use in grepping checks.
+cmarker=::: # comment marker
+pmarker=%%% # plain maker
+
+cat > pass.test <<END
+#! /bin/sh -e
+echo 1..1
+echo "$pmarker pass $pmarker" >&2
+echo "# $cmarker pass $cmarker" >&2
+echo "ok 1"
+END
+
+cat > skip.test <<END
+#! /bin/sh -e
+echo 1..1
+echo "$pmarker skip $pmarker"
+echo "# $cmarker skip $cmarker"
+echo "ok 1 # SKIP"
+END
+
+cat > xfail.test <<END
+#! /bin/sh -e
+echo 1..1
+echo "$pmarker xfail $pmarker" >&2
+echo "# $cmarker xfail $cmarker" >&2
+echo "not ok 1 # TODO"
+END
+
+cat > fail.test <<END
+#! /bin/sh -e
+echo 1..1
+echo "$pmarker fail $pmarker"
+echo "# $cmarker fail $cmarker"
+echo "not ok 1"
+END
+
+cat > xpass.test <<END
+#! /bin/sh -e
+echo 1..1
+echo "$pmarker xpass $pmarker" >&2
+echo "# $cmarker xpass $cmarker" >&2
+echo "ok 1 # TODO"
+END
+
+cat > error.test <<END
+#! /bin/sh -e
+echo 1..1
+echo "$pmarker error $pmarker"
+echo "# $cmarker error $cmarker"
+echo 'Bail out!'
+END
+
+chmod a+x *.test
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+TEST_SUITE_LOG=my.log $MAKE -e check && Exit 1
+ls -l # For debugging.
+test ! -f test-suite.log
+test ! -f global.log
+test -f my.log
+st=0
+for result in pass fail xfail xpass skip error; do
+ cat $result.log # For debugging.
+ $FGREP "$pmarker $result $pmarker" $result.log || st=1
+ $FGREP "$cmarker $result $cmarker" $result.log || st=1
+done
+test $st -eq 0 || Exit 1
+cat my.log # For debugging.
+for result in fail xpass skip error; do
+ cat $result.log # For debugging.
+ $FGREP "$pmarker $result $pmarker" my.log || st=1
+ $FGREP "$cmarker $result $cmarker" my.log || st=1
+done
+test `$FGREP -c "$pmarker" my.log` -eq 4
+test `$FGREP -c "$cmarker" my.log` -eq 4
+
+# Successful test scripts shouldn't be mentioned in the global log.
+$EGREP '(^pass|[^x]pass|xfail)\.test' my.log && Exit 1
+# But failing and skipped one should.
+$FGREP 'skip.test' my.log
+$FGREP 'fail.test' my.log
+$FGREP 'xpass.test' my.log
+$FGREP 'error.test' my.log
+
+touch error2.log test-suite.log global.log
+TEST_SUITE_LOG=my.log $MAKE -e mostlyclean
+ls -l # For debugging.
+test ! -f my.log
+test ! -f pass.log
+test ! -f fail.log
+test ! -f xfail.log
+test ! -f xpass.log
+test ! -f skip.log
+test ! -f error.log
+test -f error2.log
+test -f test-suite.log
+test -f global.log
+
+rm -f *.log
+
+VERBOSE=yes $MAKE check >stdout && { cat stdout; Exit 1; }
+cat stdout
+cat global.log
+test ! -f my.log
+test ! -f test-suite.log
+# Check that VERBOSE causes the global testsuite log to be
+# emitted on stdout.
+out=`cat stdout`
+log=`cat global.log`
+case $out in *"$log"*) ;; *) Exit 1;; esac
+
+touch error2.log test-suite.log my.log
+$MAKE mostlyclean
+ls -l # For debugging.
+test ! -f global.log
+test ! -f pass.log
+test ! -f fail.log
+test ! -f xfail.log
+test ! -f xpass.log
+test ! -f skip.log
+test ! -f error.log
+test -f error2.log
+test -f test-suite.log
+test -f my.log
+
+rm -f *.log
+
+
+:
diff --git a/tests/tap-merge-stdout-stderr.test
b/tests/tap-merge-stdout-stderr.test
new file mode 100755
index 0000000..e73f0ec
--- /dev/null
+++ b/tests/tap-merge-stdout-stderr.test
@@ -0,0 +1,85 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support:
+# - The Automake TAP driver has an option that instruct it to read TAP
+# input also from the stderr of the test command, not only its stdout.
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+AM_TEST_LOG_DRIVER_FLAGS = --comments --merge
+TESTS = all.test
+END
+
+: > all.test # Updated later.
+chmod a+x all.test
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+cat > all.test <<END
+#!/bin/sh
+echo 1..4
+echo ok 1 >&2
+echo ok 2
+echo "not ok 3 # TODO" >&2
+echo "ok 4 # SKIP"
+echo "# foo foo foo" >&2
+END
+
+$MAKE check >stdout || { cat stdout; Exit 1; }
+cat stdout
+
+count_test_results total=4 pass=2 fail=0 xpass=0 xfail=1 skip=1 error=0
+grep '^# all\.test: foo foo foo' stdout
+
+cat > all.test <<END
+#!/bin/sh
+echo 1..1
+echo ok 1
+echo 'Bail out!' >&2
+END
+
+$MAKE check >stdout && { cat stdout; Exit 1; }
+cat stdout
+
+count_test_results total=2 pass=1 fail=0 xpass=0 xfail=0 skip=0 error=1
+
+# See that the option `--no-merge' can override the effect of `--merge'.
+
+TEST_LOG_DRIVER_FLAGS=--no-merge $MAKE -e check >stdout \
+ || { cat stdout; Exit 1; }
+cat stdout
+
+count_test_results total=1 pass=1 fail=0 xpass=0 xfail=0 skip=0 error=0
+
+
+
+:
diff --git a/tests/tap-more.test b/tests/tap-more.test
new file mode 100755
index 0000000..72fcaac
--- /dev/null
+++ b/tests/tap-more.test
@@ -0,0 +1,160 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# More on TAP support:
+# - more than one TAP-generating test script in $(TESTS)
+# - VPATH builds
+# - use with parallel make (if supported)
+# - basic use of diagnostic messages (lines beginning with "#")
+# - flags for TAP driver defined through AC_SUBST in configure.ac
+# - messages generated by the testsuite harness reference the
+# correct test script(s)
+# - "make distcheck" works
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_SUBST([AM_TEST_LOG_DRIVER_FLAGS], ['--comments'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TESTS = 1.test 2.test 3.test
+EXTRA_DIST = $(TESTS) tap-driver
+END
+
+cat > 1.test <<'END'
+#! /bin/sh
+echo 1..2
+echo ok 1 - mu
+if test -f not-skip; then
+ echo "not ok 2 zardoz"
+else
+ echo "ok 2 zardoz # SKIP"
+fi
+END
+
+cat > 2.test <<'END'
+#! /bin/sh
+echo 1..3
+echo "ok"
+echo "not ok # TODO not implemented"
+echo "ok 3"
+END
+
+cat > 3.test <<END
+#! /bin/sh
+echo 1..1
+echo ok - blah blah blah
+echo '# Some diagnostic'
+if test -f bail-out; then
+ echo 'Bail out! Kernel Panic'
+else
+ :
+fi
+END
+
+chmod a+x [123].test
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+# Try a VPATH and by default serial build first, and then an in-tree
+# and by default parallel build.
+for try in 0 1; do
+
+ if test $try -eq 0; then
+ # VPATH serial build.
+ mkdir build
+ cd build
+ srcdir=..
+ run_make=$MAKE
+ elif test $try -eq 1; then
+ # In-tree parallel build.
+ srcdir=.
+ case $MAKE in
+ *\ -j*)
+ # Degree of parallelism already specified by the user: do
+ # not override it.
+ run_make=$MAKE
+ ;;
+ *)
+ # Some make implementations (e.g., HP-UX) don't grok `-j',
+ # some require no space between `-j' and the number of jobs
+ # (e.g., older GNU make versions), and some *do* require a
+ # space between `-j' and the number of jobs (e.g., Solaris
+ # dmake). We need a runtime test to see what works.
+ echo 'all:' > Makefile
+ for run_make in "$MAKE -j3" "$MAKE -j 3" "$MAKE"; do
+ $run_make && break
+ done
+ rm -f Makefile
+ ;;
+ esac
+ else
+ fatal_ "internal error, invalid value of '$try' for \$try"
+ fi
+
+ $srcdir/configure
+ ls -l # For debugging.
+
+ # Success.
+ $run_make check >stdout || { cat stdout; Exit 1; }
+ cat stdout
+ count_test_results total=6 pass=4 fail=0 xpass=0 xfail=1 skip=1 error=0
+ grep '^PASS: 1\.test 1 - mu$' stdout
+ grep '^SKIP: 1\.test 2 zardoz # SKIP$' stdout
+ test `$FGREP -c '1.test' stdout` -eq 2
+ grep '^PASS: 2\.test 1$' stdout
+ grep '^XFAIL: 2\.test 2 # TODO not implemented$' stdout
+ grep '^PASS: 2\.test 3$' stdout
+ test `$FGREP -c '2.test' stdout` -eq 3
+ grep '^PASS: 3\.test 1 - blah blah blah$' stdout
+ grep '^# 3\.test: Some diagnostic$' stdout
+ test `$FGREP -c '3.test' stdout` -eq 2
+
+ # Failure.
+ : > not-skip
+ : > bail-out
+ $run_make check >stdout && { cat stdout; Exit 1; }
+ cat stdout
+ count_test_results total=7 pass=4 fail=1 xpass=0 xfail=1 skip=0 error=1
+ grep '^PASS: 1\.test 1 - mu$' stdout
+ grep '^FAIL: 1\.test 2 zardoz$' stdout
+ test `$FGREP -c '1.test' stdout` -eq 2
+ grep '^PASS: 2\.test 1$' stdout
+ grep '^XFAIL: 2\.test 2 # TODO not implemented$' stdout
+ grep '^PASS: 2\.test 3$' stdout
+ test `$FGREP -c '2.test' stdout` -eq 3
+ grep '^PASS: 3\.test 1 - blah blah blah$' stdout
+ grep '^# 3\.test: Some diagnostic$' stdout
+ grep '^ERROR: 3\.test - Bail out! Kernel Panic$' stdout
+ test `$FGREP -c '3.test' stdout` -eq 3
+
+ cd $srcdir
+
+done
+
+$MAKE distcheck
+
+:
diff --git a/tests/tap-more2.test b/tests/tap-more2.test
new file mode 100755
index 0000000..810f9b8
--- /dev/null
+++ b/tests/tap-more2.test
@@ -0,0 +1,93 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# More on TAP support:
+# - more LOG_COMPILER at once for TAP tests
+# - binary programs in $(TESTS)
+# - interactions with `check_*' variables
+
+required='cc native'
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_PROG_CC
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_EXTENSIONS = .sh .tap
+
+tap_driver = $(PERL) $(srcdir)/tap-driver
+
+LOG_DRIVER = $(tap_driver)
+SH_LOG_DRIVER = $(tap_driver)
+TAP_LOG_DRIVER = $(tap_driver)
+
+TAP_LOG_COMPILER = cat
+SH_LOG_COMPILER = $(SHELL)
+LOG_COMPILER =
+
+EXTRA_DIST = baz.tap
+
+check_SCRIPTS = bar.sh
+bar.sh: Makefile
+ echo '#!/bin/sh' > address@hidden
+ echo 'echo 1..1' >> address@hidden
+ echo 'echo "not ok 1 # TODO"' >> address@hidden
+ chmod a-w address@hidden && mv -f address@hidden $@
+CLEANFILES = bar.sh
+
+check_PROGRAMS = foo-test
+foo_test_SOURCES = foo.c
+
+TESTS = foo-test $(check_SCRIPTS) baz.tap
+
+EXTRA_DIST += tap-driver
+END
+
+cat > foo.c <<'END'
+#include <stdio.h>
+int main (void)
+{
+ printf ("1..1\n");
+ printf ("ok 1\n");
+ return 0;
+}
+END
+
+cat > baz.tap << 'END'
+1..1
+ok 1 # SKIP
+END
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+for target in check distcheck; do
+ $MAKE $target >stdout || { cat stdout; Exit 1; }
+ cat stdout
+ count_test_results total=3 pass=1 fail=0 xpass=0 xfail=1 skip=1 error=0
+done
+
+:
diff --git a/tests/tap-no-merge-stdout-stderr.test
b/tests/tap-no-merge-stdout-stderr.test
new file mode 100755
index 0000000..c2995b2
--- /dev/null
+++ b/tests/tap-no-merge-stdout-stderr.test
@@ -0,0 +1,66 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support:
+# - By default, TAP input is only from the stdout (and not the stderr)
+# of the test command.
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TEST_LOG_DRIVER_FLAGS = --comments
+TESTS = all.test
+END
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+cat > all.test <<END
+#!/bin/sh
+echo 1..2
+echo 'not ok' >&2
+echo 'not ok 1 # TODO' >&2
+echo 'ok 1'
+echo '# foo foo foo'
+echo '# bar bar bar' >&2
+echo 'ok 44 # TODO' >&2
+echo 'Bail out!' >&2
+echo 'ok 2 # SKIP'
+END
+
+chmod a+x all.test
+
+$MAKE check >stdout || { cat stdout; Exit 1; }
+cat stdout
+
+count_test_results total=2 pass=1 fail=0 xpass=0 xfail=0 skip=1 error=0
+$FGREP 'foo foo foo' stdout
+$FGREP 'bar bar bar' stdout && Exit 1
+
+:
diff --git a/tests/tap-no-spurious-summary.test
b/tests/tap-no-spurious-summary.test
new file mode 100755
index 0000000..4c358da
--- /dev/null
+++ b/tests/tap-no-spurious-summary.test
@@ -0,0 +1,65 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support:
+# - `:test-results:' directives in test scripts' output doesn't
+# originate spurious results in the testsuite summary
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TEST_LOG_COMPILER = cat
+TESTS = all.test
+END
+
+cat > all.test <<'END'
+1..1
+:test-result: PASS
+:test-result: FAIL
+:test-result: XPASS
+:test-result: XFAIL
+:test-result: SKIP
+:test-result: ERROR
+:test-result: UNKNOWN
+ok 1
+END
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+$MAKE check >stdout || { cat stdout; Exit 1; }
+cat stdout
+
+count_test_results total=1 pass=1 fail=0 xpass=0 xfail=0 skip=0 error=0
+
+for result in PASS FAIL XPASS XFAIL SKIP ERROR UNKNOWN; do
+ grep "^ *:test-result: $result$" all.log
+done
+
+:
diff --git a/tests/tap-no-spurious.test b/tests/tap-no-spurious.test
new file mode 100755
index 0000000..cfe7dbd
--- /dev/null
+++ b/tests/tap-no-spurious.test
@@ -0,0 +1,114 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support:
+# - don't spuriously recognize lines that are "almost" TAP lines as
+# real TAP lines
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TEST_LOG_COMPILER = cat
+TESTS = all.test
+END
+
+echo 1..5 > all.test
+
+# The only recognized directives are "TODO" and "SKIP".
+# So the following should count as passed tests.
+cat >> all.test <<END
+ok 1 # XFAIL
+ok 2 # SKIPPED
+ok 3 # TO DO
+ok 4 # TODOALL
+ok 5 # FIXME
+END
+
+# According to documentation of Test::Harness::TAP(3):
+#
+# ``Lines written to standard output matching /^(not )?ok\b/
+# must be interpreted as test lines. All other lines must
+# not be considered test output.''
+
+cat >> all.test <<END
+ ok
+ ok 1
+${tab}ok
+${tab}ok 1
+ok1
+ok23
+ not ok
+ not ok 1
+${tab}not ok
+${tab}not ok 1
+not ok1
+not ok23
+notok
+notok 1
+not${tab}ok
+not${tab}ok 1
+not ok
+not ok 1
+no ok
+no ok 1
+# ok
+# not ok
+# ok 1
+# not ok 1
+#ok
+#not ok
+#ok 1
+#not ok 1
+END
+
+# The prove(1) utility doesn't bail out on these, so our driver
+# shouldn't either.
+cat >> all.test <<'END'
+bailout
+bailout!
+bail out
+bail out!
+Bailout
+Bailout!
+Bail out
+ Bail out
+ Bail out!
+#Bail out!
+# Bail out!
+END
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+$MAKE check >stdout || { cat stdout; Exit 1; }
+cat stdout
+
+count_test_results total=5 pass=5 fail=0 xpass=0 xfail=0 skip=0 error=0
+
+:
diff --git a/tests/tap-not-ok-skip.test b/tests/tap-not-ok-skip.test
new file mode 100755
index 0000000..57b8d7a
--- /dev/null
+++ b/tests/tap-not-ok-skip.test
@@ -0,0 +1,57 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support:
+# - a "not ok # SKIP" line should count as a failure, for consistency
+# with the prove(1) utility.
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TEST_LOG_COMPILER = cat
+TESTS = all.test
+END
+
+cat > all.test <<'END'
+1..4
+not ok # SKIP
+not ok 2 # SKIP
+not ok - foo # SKIP
+not ok 4 - bar # SKIP
+END
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+$MAKE check >stdout && { cat stdout; Exit 1; }
+cat stdout
+
+count_test_results total=4 pass=0 fail=4 skip=0 xpass=0 xfail=0 error=0
+
+:
diff --git a/tests/tap-numeric-description.test
b/tests/tap-numeric-description.test
new file mode 100755
index 0000000..67b3733
--- /dev/null
+++ b/tests/tap-numeric-description.test
@@ -0,0 +1,86 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP result lines whose description is a number.
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TEST_LOG_COMPILER = cat
+TESTS = all.test
+END
+
+# Some random numbers to be used as test names.
+# These are selected so that $n != n for every n >= 1.
+set 5 7 45 11 7 3 6 9 1000 1
+# $1 $2 $3 $4 $5 $6 $7 $8 $9 $10
+
+cat > all.test <<END
+1..10
+ok 1 ${1}
+ok - ${2}
+not ok 3 ${3}
+not ok - ${4}
+ok 5 ${5} # SKIP
+ok - ${6} # SKIP
+not ok 7 ${7} # TODO
+not ok - ${8} # TODO
+ok 9 ${9} # TODO
+ok - ${10} # TODO
+END
+
+cat > exp <<END
+PASS: all.test 1 ${1}
+PASS: all.test 2 - ${2}
+FAIL: all.test 3 ${3}
+FAIL: all.test 4 - ${4}
+SKIP: all.test 5 ${5} # SKIP
+SKIP: all.test 6 - ${6} # SKIP
+XFAIL: all.test 7 ${7} # TODO
+XFAIL: all.test 8 - ${8} # TODO
+XPASS: all.test 9 ${9} # TODO
+XPASS: all.test 10 - ${10} # TODO
+END
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+$MAKE check >stdout && { cat stdout; Exit 1; }
+cat stdout
+
+count_test_results total=10 pass=2 fail=2 xpass=2 xfail=2 skip=2 error=0
+
+$FGREP ': all.test' stdout > got
+
+cat exp
+cat got
+diff exp got
+
+
+:
diff --git a/tests/tap-out-of-order.test b/tests/tap-out-of-order.test
new file mode 100755
index 0000000..b1821cd
--- /dev/null
+++ b/tests/tap-out-of-order.test
@@ -0,0 +1,94 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support:
+# - out-of-order test results
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TEST_LOG_COMPILER = cat
+TESTS = a.test b.test c.test d.test
+END
+
+cat > a.test <<END
+1..3
+ok
+not ok 2 # TODO
+ok 71
+END
+
+cat > b.test <<END
+1..5
+ok 1
+ok 2
+ok 4 foo # SKIP
+not ok 2 # TODO
+ok 5
+END
+
+cat > c.test <<END
+1..4
+ok 1
+ok foo
+ok 4 - bar
+not ok 3
+END
+
+# This is documented as an "test out of sequence" error in the
+# TAP::Parser documentation. Keep it in sync with the example
+# there.
+cat > d.test <<'END'
+1..5
+ok 1
+ok 2
+ok 15
+ok 16
+ok 17
+END
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+$MAKE check >stdout && { cat stdout; Exit 1; }
+cat stdout
+
+count_test_results total=18 pass=11 fail=0 xpass=0 xfail=1 skip=0 error=6
+test `$FGREP -c ': a.test' stdout` -eq 3
+test `$FGREP -c ': b.test' stdout` -eq 5
+test `$FGREP -c ': c.test' stdout` -eq 4
+test `$FGREP -c ': d.test' stdout` -eq 5
+
+grep '^ERROR: a\.test - test out of order (expecting 3, got 71)$'
+grep '^ERROR: b\.test - test out of order (expecting 3, got 4)$'
+grep '^ERROR: b\.test - test out of order (expecting 4, got 2)$'
+grep '^ERROR: c\.test - test out of order (expecting 3, got 4)$'
+grep '^ERROR: d\.test - test out of order (expecting 3, got 15)$'
+
+:
diff --git a/tests/tap-passthrough.test b/tests/tap-passthrough.test
new file mode 100755
index 0000000..75df5aa
--- /dev/null
+++ b/tests/tap-passthrough.test
@@ -0,0 +1,188 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support:
+# - all input (valid TAP lines, invalid TAP lines, non-TAP lines)
+# are passed through in the log file
+# - TAP parse errors are reported in the log file too.
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TEST_LOG_COMPILER = cat
+TESTS =
+END
+
+weirdchars=\''"address@hidden&()[]<>#;,:.^?*/'
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+#
+# Only successful tests.
+#
+
+cat > ok.test <<END
+1..6
+TAP plan in the previous line.
+ok${tab}
+ok 2
+ok - foo
+ok 4 - x
+ This is not a TAP line, but should still be copied in the log file!
+# diagnostic
+not ok # TODO low priority
+ok # SKIP who cares?
+This is not in test-suite.log, since ok.test does not fail.
+$weirdchars
+END
+
+st=0
+TESTS=ok.test $MAKE -e check || st=$?
+cat ok.log
+cat test-suite.log
+test $st -eq 0 || Exit 1
+
+for rx in \
+ '1\.\.6' \
+ 'TAP plan in the previous line\.' \
+ "ok${tab}" \
+ 'ok 2' \
+ 'ok - foo' \
+ 'ok 4 - x' \
+ ' This is not a TAP line, but should still be copied in the log file!' \
+ '# diagnostic' \
+ 'not ok # TODO low priority' \
+ 'ok # SKIP who cares?' \
+ 'This is not in test-suite\.log, since ok\.test does not fail\.' \
+; do
+ grep "^$rx$" ok.log
+done
+$FGREP "$weirdchars" ok.log
+
+$FGREP 'ok.test' test-suite.log && Exit 1
+$FGREP 'not in test-suite.log' test-suite.log && Exit 1
+
+#
+# Mixed failing/successful tests.
+#
+
+cat > ok.test <<END
+1..1
+ok
+local only
+END
+
+cat > ko.test <<END
+1..5
+foo foo foo
+ok${tab}
+ok 2
+not ok - foo
+not ok 4 - x
+# diagnostic ko
+ bar${tab}bar${tab}bar
+ok # TODO dunno
+$weirdchars
+END
+
+cat > bail.test <<END
+Bail out! Test is taking too long!
+END
+
+cat > skip.test <<END
+1..0 # Skipped: WWW::Mechanize not installed
+END
+
+cat > err.test <<END
+1..3
+ok 1
+Invalid test count
+ok 23
+Misplaced plan
+1..3
+ok
+Extra test
+ok
+Last line
+END
+
+st=0
+TESTS='ok.test ko.test bail.test skip.test err.test' $MAKE -e check || st=$?
+cat ok.log
+cat ko.log
+cat bail.log
+cat skip.log
+cat err.log
+cat test-suite.log
+test $st -gt 0 || Exit 1
+
+grep '^local only$' ok.log
+grep 'local only' test-suite.log && Exit 1
+
+for rx in \
+ '1\.\.5' \
+ 'foo foo foo' \
+ "ok${tab}" \
+ 'ok 2' \
+ 'not ok - foo' \
+ 'not ok 4 - x' \
+ '# diagnostic ko' \
+ " bar${tab}bar${tab}bar" \
+ 'ok # TODO dunno' \
+; do
+ grep "^$rx$" ko.log
+ grep "^$rx$" test-suite.log
+done
+$FGREP "$weirdchars" ko.log
+$FGREP "$weirdchars" test-suite.log
+
+for log in bail.log test-suite.log; do
+ grep '^Bail out! Test is taking too long!$' $log
+done
+for log in skip.log test-suite.log; do
+ grep '^1\.\.0 # Skipped: WWW::Mechanize not installed$' $log
+done
+
+for rx in \
+ '^Invalid test count$' \
+ '^Misplaced plan$' \
+ '^Extra test$' \
+ '^Last line$' \
+ '^ok 23$' \
+ '[tT]oo many tests run.*expected 3, got 4' \
+ '[oO]ut of order.*expecting 2, got 23'
+; do
+ grep "$rx" err.log
+ grep "$rx" test-suite.log
+done
+test `grep -c '^1\.\.3$' err.log` -eq 2
+test `grep -c '^1\.\.3$' test-suite.log` -eq 2
+
+:
diff --git a/tests/tap-plan-errors.test b/tests/tap-plan-errors.test
new file mode 100755
index 0000000..e8d0935
--- /dev/null
+++ b/tests/tap-plan-errors.test
@@ -0,0 +1,168 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support: the following are flagged as errors:
+# - unmatched test plan
+# - misplaced test plan
+# - multiple test plans
+# - missing test plan
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TEST_LOG_COMPILER = cat
+TESTS = foo.test
+END
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+my_check ()
+{
+ cat > foo.test
+ $MAKE check >stdout && { cat stdout; Exit 1; }
+ cat stdout
+ count_test_results "$@"
+}
+
+my_check pass=1 fail=0 xpass=0 xfail=0 skip=0 error=1 <<END
+1..2
+ok 1
+END
+grep '^ERROR: foo\.test - too few tests run (expected 2, got 1)$' stdout
+
+my_check pass=2 fail=0 xpass=0 xfail=0 skip=0 error=1 <<END
+ok 1
+ok 2
+not ok 3 # TODO
+1..12
+END
+grep '^ERROR: foo\.test - too few tests run (expected 12, got 3)$' stdout
+
+my_check pass=0 fail=0 xpass=0 xfail=0 skip=0 error=1 <<END
+1..1
+END
+grep '^ERROR: foo\.test - too few tests run (expected 1, got 0)$' stdout
+
+my_check pass=2 fail=0 xpass=0 xfail=0 skip=0 error=1 <<END
+1..1
+ok 1
+ok 2
+END
+grep '^ERROR: foo\.test - too many tests run (expected 1, got 2)$' stdout
+
+my_check pass=4 fail=0 xpass=0 xfail=0 skip=5 error=1 <<END
+1..4
+ok 1
+ok 2
+ok 3
+ok 4
+ok 5
+ok 6
+ok 7
+END
+grep '^ERROR: foo\.test - too many tests run (expected 4, got 7)$' stdout
+
+my_check pass=2 fail=0 xpass=0 xfail=1 skip=0 error=1 <<END
+ok 1
+ok 2
+not ok 3 # TODO
+1..2
+END
+grep '^ERROR: foo\.test - too many tests run (expected 2, got 3)$' stdout
+
+# Leading blank line is meant.
+my_check pass=2 fail=0 xpass=0 xfail=0 skip=0 error=1 <<END
+
+1..2
+ok 1
+ok 2
+END
+grep '^ERROR: foo\.test - test plan in middle of output$' stdout
+
+# Trailing blank line is meant.
+my_check pass=2 fail=0 xpass=0 xfail=0 skip=0 error=1 <<END
+ok 1
+ok 2
+1..2
+
+END
+grep '^ERROR: foo\.test - test plan in middle of output$' stdout
+
+# In the next three checks, the two plans are equal -- this is deliberate.
+
+my_check pass=2 fail=0 xpass=0 xfail=0 skip=0 error=1 <<END
+1..1
+1..1
+ok 1
+END
+grep '^ERROR: foo\.test - multiple test plans$' stdout
+
+my_check pass=2 fail=0 xpass=0 xfail=0 skip=0 error=1 <<END
+ok 1
+1..1
+1..1
+END
+grep '^ERROR: foo\.test - multiple test plans$' stdout
+
+my_check pass=2 fail=0 xpass=0 xfail=0 skip=0 error=1 <<END
+1..2
+ok 1
+ok 2
+1..2
+END
+grep '^ERROR: foo\.test - multiple test plans$' stdout
+
+my_check pass=1 fail=0 xpass=0 xfail=0 skip=0 error=1 <<END
+ok 1
+END
+grep '^ERROR: foo\.test - missing test plans$' stdout
+
+my_check pass=0 fail=0 xpass=0 xfail=0 skip=0 error=1 </dev/null
+grep '^ERROR: foo\.test - missing test plans$' stdout
+
+# Check error message in corner cases (1).
+my_check pass=2 fail=0 xpass=0 xfail=0 skip=0 error=1 <<END
+1..2
+ok 1
+1..2
+ok 2
+END
+grep '^ERROR: foo\.test - test plan in middle of output$' stdout
+
+# Check error message in corner cases (2).
+my_check pass=2 fail=0 xpass=0 xfail=0 skip=0 error=1 <<END
+ok 1
+1..2
+ok 2
+1..2
+END
+grep '^ERROR: foo\.test - test plan in middle of output$' stdout
+
+:
diff --git a/tests/tap-plan.test b/tests/tap-plan.test
new file mode 100755
index 0000000..10fa377
--- /dev/null
+++ b/tests/tap-plan.test
@@ -0,0 +1,69 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support:
+# - test scripts with the test plan at the beginning
+# - test scripts with the test plan at the end
+# - test scripts without a test plan
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TEST_LOG_COMPILER = cat
+TESTS = top.test bot.test
+END
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+cat > top.test <<END
+1..3
+ok 1
+ok 2 # SKIP
+ok 3
+# a trailing comment
+END
+
+cat > bot.test <<END
+ok 1
+# a comment
+ok 2
+not ok 3 # TODO
+# another comment
+ok
+1..4
+END
+
+# Check that the plans doesn't cause any problem
+
+$MAKE check >stdout || { cat stdout; Exit 1; }
+cat stdout
+count_test_results total=7 pass=5 xfail=1 skip=1 fail=0 xpass=0 error=0
+
+:
diff --git a/tests/tap-realtime.test b/tests/tap-realtime.test
new file mode 100755
index 0000000..f5b9496
--- /dev/null
+++ b/tests/tap-realtime.test
@@ -0,0 +1,96 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support:
+# - testsuite progress on console should happen mostly "in real time";
+# i.e., it's not acceptable for the driver to wait the end of the
+# script to start displaying results from it.
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+# Check that we have a working expect program.
+cat >expect-check <<'END'
+eval spawn $env(SHELL) -c "echo ok"
+expect "^ok$"
+END
+expect -f expect-check || {
+ echo "$me: failed to find a working expect program" >&2
+ Exit 77
+}
+rm -f expect-check
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TESTS = all.test
+END
+
+cat > all.test <<'END'
+#! /bin/sh
+echo 1..3
+echo ok 1
+: > milestone-1
+sleep '10' # FIXME: quoting required to placate maintainer-check
+echo ok 2
+: > milestone-2
+sleep '10' # FIXME: quoting required to placate maintainer-check
+echo ok 3
+END
+
+chmod a+x all.test
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+cat >expect-make <<'END'
+eval spawn $env(MAKE) -e check
+expect eof
+END
+
+# Expect should simulate a tty as stdout, which should ensure a
+# line-buffered output.
+MAKE=$MAKE expect -f expect-make > stdout &
+
+while test ! -f milestone-1; do sleep '1'; done
+cat stdout
+grep '^PASS: all\.test 1$' stdout
+test `$FGREP -c ': all.test'` -eq 1
+
+while test ! -f milestone-2; do sleep '1'; done
+cat stdout
+grep '^PASS: all\.test 1$' stdout
+grep '^PASS: all\.test 2$' stdout
+test `$FGREP -c ': all.test'` -eq 2
+
+wait
+cat stdout
+grep '^PASS: all\.test 1$' stdout
+grep '^PASS: all\.test 2$' stdout
+grep '^PASS: all\.test 3$' stdout
+test `$FGREP -c ': all.test'` -eq 2
+
+:
diff --git a/tests/tap-recheck-logs.test b/tests/tap-recheck-logs.test
new file mode 100755
index 0000000..a3484bb
--- /dev/null
+++ b/tests/tap-recheck-logs.test
@@ -0,0 +1,132 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support:
+# - RECHECK_LOGS
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TEST_LOG_COMPILER = cat
+TESTS = foo.test bar.test baz.test
+baz.log: zardoz
+END
+
+: > zardoz
+
+cat > foo.test <<'END'
+1..2
+ok 1
+ok 2
+END
+
+cat > bar.test <<'END'
+1..1
+not ok 1
+END
+
+cat > baz.test <<'END'
+1..1
+Bail out!
+END
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE -a
+
+./configure
+
+# Even the tests that are not re-run should contribute to the testsuite
+# summary when obtained by "make check RECHECK_LOGS=".
+grep_summary ()
+{
+ grep '^# TOTAL: *4$' stdout
+ grep '^# PASS: *2$' stdout
+ grep '^# XPASS: *0$' stdout
+ grep '^# FAIL: *1$' stdout
+ grep '^# XFAIL: *0$' stdout
+ grep '^# SKIP: *0$' stdout
+ grep '^# ERROR: *1$' stdout
+}
+
+$MAKE -e check && Exit 1
+test -f foo.log
+test -f bar.log
+test -f baz.log
+
+rm -f foo.log bar.log
+
+env RECHECK_LOGS= $MAKE -e check > stdout && { cat stdout; Exit 1; }
+cat stdout
+test -f foo.log
+test -f bar.log
+grep '^PASS: foo\.test 1$' stdout
+grep '^PASS: foo\.test 2$' stdout
+grep '^FAIL: bar\.test 1$' stdout
+grep 'baz\.test' stdout && Exit 1
+grep_summary
+
+$sleep
+touch foo.test
+# We re-run only a successful test, but the tests that failed in the
+# previous run should still be taken into account, and cause an overall
+# failure.
+env RECHECK_LOGS= $MAKE -e check > stdout && { cat stdout; Exit 1; }
+cat stdout
+grep '^PASS: foo\.test 1$' stdout
+grep '^PASS: foo\.test 2$' stdout
+grep 'ba[rz]\.test' stdout && Exit 1
+is_newest foo.log foo.test
+grep_summary
+
+$sleep
+touch zardoz
+env RECHECK_LOGS= $MAKE -e check > stdout && { cat stdout; Exit 1; }
+cat stdout
+grep '^ERROR: baz\.test' stdout
+$EGREP '(foo|bar)\.test' stdout && Exit 1
+is_newest baz.log zardoz
+grep_summary
+
+# Now, explicitly retry with all test logs already updated, and ensure
+# that the summary is still displayed.
+env RECHECK_LOGS= $MAKE -e check > stdout && { cat stdout; Exit 1; }
+cat stdout
+$EGREP '(foo|bar|baz)\.test' stdout && Exit 1
+grep_summary
+
+# The following should re-run foo.test (and only foo.test), even if its
+# log file is up-to-date.
+: > older
+env RECHECK_LOGS=foo.log $MAKE -e check > stdout && { cat stdout; Exit 1; }
+cat stdout
+grep '^PASS: foo\.test 1$' stdout
+grep '^PASS: foo\.test 2$' stdout
+grep 'ba[rz]\.test' stdout && Exit 1
+is_newest foo.log older
+grep_summary
+
+:
diff --git a/tests/tap-recheck.test b/tests/tap-recheck.test
new file mode 100755
index 0000000..27893c4
--- /dev/null
+++ b/tests/tap-recheck.test
@@ -0,0 +1,215 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# Test the 'recheck' target for TAP test protocol.
+# Keep in sync with 'test-driver-custom-multitest-recheck.test'.
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TESTS = a.test b.test c.test d.test
+END
+
+cat > a.test << 'END'
+#! /bin/sh
+echo 1..2
+echo ok 1
+echo ok 2
+: > a.run
+END
+
+cat > b.test << 'END'
+#! /bin/sh
+: > b.run
+echo 1..2
+echo ok 1
+if test -f b.ok; then
+ echo ok 2
+else
+ echo 'Bail out!'
+fi
+END
+
+cat > c.test << 'END'
+#! /bin/sh
+echo 1..2
+if test -f c.pass; then
+ echo 'ok - c is ok :-)'
+else
+ echo 'not ok - c is ko :-('
+fi
+if test -f c.xfail; then
+ echo 'not ok 2 # TODO'
+else
+ echo 'ok 2 # TODO'
+fi
+echo 'not ok 3 - blah blah # TODO need better diagnostic'
+: > c.run
+END
+
+cat > d.test << 'END'
+#! /bin/sh
+echo ok 1 '# SKIP: who cares ...'
+(. ./d.extra) || echo 'not ok 2 - d.extra failed'
+: > d.run
+END
+
+chmod a+x *.test
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+do_recheck ()
+{
+ case $* in
+ --fail) on_bad_rc='&&';;
+ --pass) on_bad_rc='||';;
+ *) fatal_ "invalid usage of function 'do_recheck'";;
+ esac
+ rm -f *.run
+ eval "\$MAKE recheck >stdout $on_bad_rc { cat stdout; ls -l; Exit 1; }; :"
+ cat stdout; ls -l
+}
+
+for vpath in : false; do
+ if $vpath; then
+ mkdir build
+ cd build
+ srcdir=..
+ else
+ srcdir=.
+ fi
+
+ $srcdir/configure
+
+ : A "make recheck" in a clean tree should run no tests.
+ do_recheck --pass
+ cat test-suite.log
+ test ! -r a.run
+ test ! -r a.log
+ test ! -r b.run
+ test ! -r b.log
+ test ! -r c.run
+ test ! -r c.log
+ test ! -r d.run
+ test ! -r d.log
+ count_test_results total=0 pass=0 fail=0 xpass=0 xfail=0 skip=0 error=0
+
+ : Run the tests for the first time.
+ $MAKE check >stdout && { cat stdout; Exit 1; }
+ cat stdout
+ ls -l
+ # All the test scripts should have run.
+ test -f a.run
+ test -f b.run
+ test -f c.run
+ test -f d.run
+ count_test_results total=9 pass=3 fail=2 xpass=1 xfail=1 skip=1 error=1
+ head -1 *.log
+
+ : Let us make b.test pass.
+ echo OK > b.ok
+ do_recheck --fail
+ # a.test has been successful the first time, so no need to re-run it.
+ # Similar considerations apply to similar checks, below.
+ test ! -r a.run
+ test -f b.run
+ test -f c.run
+ test -f d.run
+ count_test_results total=7 pass=2 fail=2 xpass=1 xfail=1 skip=1 error=0
+
+ : Let us make the first part of c.test pass.
+ echo OK > c.pass
+ do_recheck --fail
+ test ! -r a.run
+ test ! -r b.run
+ test -f c.run
+ test -f d.run
+ count_test_results total=5 pass=1 fail=1 xpass=1 xfail=1 skip=1 error=0
+
+ : Let us make also the second part of c.test pass.
+ echo KO > c.xfail
+ do_recheck --fail
+ test ! -r a.run
+ test ! -r b.run
+ test -f c.run
+ test -f d.run
+ count_test_results total=5 pass=1 fail=1 xpass=0 xfail=2 skip=1 error=0
+
+ : Nothing changed, so only d.test should be run.
+ for i in 1 2; do
+ do_recheck --fail
+ test ! -r a.run
+ test ! -r b.run
+ test ! -r c.run
+ test -f d.run
+ count_test_results total=2 pass=0 fail=1 xpass=0 xfail=0 skip=1 error=0
+ done
+
+ : Let us make d.test run more testcases, and experience _more_ failures.
+ unindent > d.extra <<'END'
+ echo 'ok # SKIP s'
+ echo 'not ok - f 1'
+ echo 'ok - p 1'
+ echo 'not ok - f 2'
+ echo 'ok # TODO xp'
+ echo 'not ok - f 3'
+ echo 'not ok - f 4'
+ # Misplaced plan
+ echo '1..4'
+ echo 'ok - p 2'
+ echo 'Bail out!'
+END
+ do_recheck --fail
+ test ! -r a.run
+ test ! -r b.run
+ test ! -r c.run
+ test -f d.run
+ count_test_results total=11 pass=2 fail=4 xpass=1 xfail=0 skip=2 error=2
+
+ : Let us finally make d.test pass.
+ echo : > d.extra
+ do_recheck --pass
+ test ! -r a.run
+ test ! -r b.run
+ test ! -r c.run
+ test -f d.run
+ count_test_results total=1 pass=0 fail=0 xpass=0 xfail=0 skip=1 error=0
+
+ : All tests have been successful or skipped, nothing should be re-run.
+ do_recheck --pass
+ test ! -r a.run
+ test ! -r b.run
+ test ! -r c.run
+ test ! -r d.run
+ count_test_results total=0 pass=0 fail=0 xpass=0 xfail=0 skip=0 error=0
+
+ cd $srcdir
+
+done
+
+:
diff --git a/tests/tap-skip-whole.test b/tests/tap-skip-whole.test
new file mode 100755
index 0000000..bd75e90
--- /dev/null
+++ b/tests/tap-skip-whole.test
@@ -0,0 +1,100 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# Basic TAP test protocol support:
+# - special plan format to skip all the tests in a script
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+weirdchars=\''"$!&()[]<>#;^?*/@%=,.:'
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TEST_LOG_COMPILER = cat
+TESTS = foo.test bar.test baz.test wget.test curl.test mu.test
+END
+
+cat > foo.test <<END
+1..0
+not ok 1
+not ok 2
+END
+
+cat > bar.test <<END
+1..0$tab $tab
+ok 1
+Bail out! some random failure
+END
+
+# It is undefined whether the comment after the plan below should
+# count as an explanation; still, the test should be skipped.
+cat > baz.test <<END
+1..0 # WWW::Mechanize not installed
+ok 1
+END
+
+cat > wget.test <<END
+1..0 # SKIP wget(1) not installed
+not ok 1 # TODO
+ok 2
+ok 3 # SKIP
+END
+
+cat > curl.test <<END
+1..0 # skip: Can't connect to gnu.org!
+not ok 1
+ok 2 # TODO
+END
+
+cat > mu.test <<END
+1..0 # Skip $weirdchars
+# Various TAP errors in here shouldn't be seen, as the test is skipped.
+1..2
+ok 1
+ok 22
+not ok 333
+1..7
+END
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+$MAKE check >stdout || { cat stdout; Exit 1; }
+cat stdout
+
+count_test_results total=6 pass=0 fail=0 xpass=0 xfail=0 skip=6 error=0
+
+grep '^SKIP: foo\.test$' stdout
+grep '^SKIP: bar\.test$' stdout
+grep '^SKIP: baz\.test' stdout # Deliberately laxer, see above for why.
+grep '^SKIP: wget\.test .* wget(1) not installed$' stdout
+grep '^SKIP: curl\.test .* Can'\''t connect to gnu\.org!$' stdout
+grep '^SKIP: mu\.test' stdout | $FGREP "$weirdchars" stdout
+test `grep -c ': .*\.test' stdout` -eq 6
+
+:
diff --git a/tests/tap-summary.test b/tests/tap-summary.test
new file mode 100755
index 0000000..649485e
--- /dev/null
+++ b/tests/tap-summary.test
@@ -0,0 +1,363 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support:
+# - testsuite summary
+# - colorized testsuite summary
+# TODO: splitting this into two distinct test scripts might be worthwhile
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat > configure.in <<END
+AC_INIT([GNU AutoTAP], [5.12], address@hidden)
+AM_INIT_AUTOMAKE([parallel-tests])
+AC_SUBST([PERL], ['$PERL'])
+AC_CONFIG_FILES([Makefile])
+AC_OUTPUT
+END
+
+cat > Makefile.stub << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TEST_LOG_COMPILER = cat
+TESTS = all.test
+END
+
+$ACLOCAL
+$AUTOCONF
+
+br='============================================================================'
+
+# The following shell variables are influential for this function:
+# - expect_failure
+# - use_colors
+do_check ()
+{
+ case $#,$1 in
+ 1,--pass) expect_failure=no;;
+ 1,--fail) expect_failure=yes;;
+ *) fatal_ "invalid usage of 'do_check'";;
+ esac
+ shift
+ cat > summary.exp
+ cat all.test
+ st=0
+ $MAKE check > stdout || st=$?
+ cat stdout
+ if test $expect_failure = yes; then
+ test $st -gt 0 || Exit 1
+ else
+ test $st -eq 0 || Exit 1
+ fi
+ $PERL -w "$testsrcdir"/extract-testsuite-summary stdout > summary.got \
+ || fatal_ "cannot extract testsuite summary"
+ cat summary.exp
+ cat summary.got
+ if test $use_colors = yes; then
+ # Use cmp, not diff, because the files might contain binary data.
+ compare=cmp
+ else
+ compare=diff
+ fi
+ $compare summary.exp summary.got || Exit 1
+}
+
+# FIXME: also enable testing of colors!
+#for use_colors in "no" "yes"; do
+for use_colors in "no"; do
+
+ cp Makefile.stub Makefile.am
+
+ if test $use_colors = yes; then
+ red='[0;31m'
+ grn='[0;32m'
+ lgn='[1;32m'
+ blu='[1;34m'
+ mgn='[0;35m'
+ brg='[1m'
+ std='[m'
+ echo AUTOMAKE_OPTIONS = color-tests >> Makefile.am
+ else
+ red= grn= lgn= blu= mgn= brg= std=
+ fi
+
+ success_header="\
+${grn}${br}${std}
+${grn}Testsuite summary for GNU AutoTAP 5.12${std}
+${grn}${br}${std}"
+
+ success_footer=${grn}${br}${std}
+
+ failure_header="\
+${red}${br}${std}
+${red}Testsuite summary for GNU AutoTAP 5.12${std}
+${red}${br}${std}"
+
+ failure_footer="\
+${red}${br}${std}
+${red}See ./test-suite.log${std}
+${red}Please report to address@hidden
+${red}${br}${std}"
+
+ $AUTOMAKE --add-missing
+
+ ./configure
+
+ # 1 pass.
+ { echo 1..1 && echo ok; } > all.test
+ do_check --pass <<END
+$success_header
+${brg}# TOTAL: 1${std}
+${grn}# PASS: 1${std}
+# SKIP: 0
+# XFAIL: 0
+# FAIL: 0
+# XPASS: 0
+# ERROR: 0
+$success_footer
+END
+
+ # 1 skip.
+ { echo 1..1 && echo 'ok # SKIP'; } > all.test
+ do_check --pass <<END
+$success_header
+${brg}# TOTAL: 1${std}
+# PASS: 0
+${blu}# SKIP: 1${std}
+# XFAIL: 0
+# FAIL: 0
+# XPASS: 0
+# ERROR: 0
+$success_footer
+END
+
+ # 1 xfail.
+ { echo 1..1 && echo 'not ok # TODO'; } > all.test
+ do_check --pass <<END
+$success_header
+${brg}# TOTAL: 1${std}
+# PASS: 0
+# SKIP: 0
+${lgn}# XFAIL: 1${std}
+# FAIL: 0
+# XPASS: 0
+# ERROR: 0
+$success_footer
+END
+
+ # 1 fail.
+ { echo 1..1 && echo not ok; } > all.test
+ do_check --fail <<END
+$failure_header
+${brg}# TOTAL: 1${std}
+# PASS: 0
+# SKIP: 0
+# XFAIL: 0
+${red}# FAIL: 1${std}
+# XPASS: 0
+# ERROR: 0
+$failure_footer
+END
+
+ # 1 xpass.
+ { echo 1..1 && echo 'ok # TODO'; } > all.test
+ do_check --fail <<END
+$failure_header
+${brg}# TOTAL: 1${std}
+# PASS: 0
+# SKIP: 0
+# XFAIL: 0
+# FAIL: 0
+${red}# XPASS: 1${std}
+# ERROR: 0
+$failure_footer
+END
+
+ # 1 hard error.
+ { echo 1..1 && echo 'Bail out!'; } > all.test
+ do_check --fail <<END
+$failure_header
+${brg}# TOTAL: 1${std}
+# PASS: 0
+# SKIP: 0
+# XFAIL: 0
+# FAIL: 0
+# XPASS: 0
+${mgn}# ERROR: 1${std}
+$failure_footer
+END
+
+ # 3 non-failing results.
+ cat > all.test <<END
+1..3
+ok
+not ok # TODO
+ok # SKIP
+END
+ do_check --pass <<END
+$success_header
+${brg}# TOTAL: 3${std}
+${grn}# PASS: 1${std}
+${blu}# SKIP: 1${std}
+${lgn}# XFAIL: 1${std}
+# FAIL: 0
+# XPASS: 0
+# ERROR: 0
+$success_footer
+END
+
+ # 1 pass, 1 skip, 1 fail.
+ cat > all.test <<END
+1..3
+ok
+ok # SKIP
+not ok
+END
+ do_check --fail <<END
+$failure_header
+${brg}# TOTAL: 3${std}
+${grn}# PASS: 1${std}
+${blu}# SKIP: 1${std}
+# XFAIL: 0
+${red}# FAIL: 1${std}
+# XPASS: 0
+# ERROR: 0
+$failure_footer
+END
+
+ # 1 pass, 1 xfail, 1 xpass.
+ cat > all.test <<END
+1..3
+ok
+ok # TODO
+not ok # TODO
+END
+ do_check --fail <<END
+$failure_header
+${brg}# TOTAL: 3${std}
+${grn}# PASS: 1${std}
+# SKIP: 0
+${lgn}# XFAIL: 1${std}
+# FAIL: 0
+${red}# XPASS: 1${std}
+# ERROR: 0
+$failure_footer
+END
+
+ # 1 skip, 1 xfail, 1 error.
+ cat > all.test <<END
+1..3
+ok # SKIP
+not ok # TODO
+Bail out!
+END
+ do_check --fail <<END
+$failure_header
+${brg}# TOTAL: 3${std}
+# PASS: 0
+${blu}# SKIP: 1${std}
+${lgn}# XFAIL: 1${std}
+# FAIL: 0
+# XPASS: 0
+${mgn}# ERROR: 1${std}
+$failure_footer
+END
+
+ # 1 of each kind
+ cat > all.test <<END
+1..6
+ok
+not ok
+ok # TODO
+not ok # TODO
+ok # SKIP
+Bail out!
+END
+ do_check --fail <<END
+$failure_header
+${brg}# TOTAL: 6${std}
+${grn}# PASS: 1${std}
+${blu}# SKIP: 1${std}
+${lgn}# XFAIL: 1${std}
+${red}# FAIL: 1${std}
+${red}# XPASS: 1${std}
+${mgn}# ERROR: 1${std}
+$failure_footer
+END
+
+ # Prepare some common data for later.
+ for i in 0 1 2 3 4 5 6 7 8 9; do
+ for j in 0 1 2 3 4 5 6 7 8 9; do
+ echo "ok"
+ echo "not ok # TODO"
+ echo "ok # SKIP"
+ done
+ done > tap
+
+ # Lots of non-failures (300 per kind).
+ (cat tap && cat tap && cat tap) > all.test
+ test `wc -l <all.test` -eq 900 || Exit 99 # Sanity check.
+ echo 1..900 >> all.test # Test plan.
+ do_check --pass <<END
+$success_header
+${brg}# TOTAL: 900${std}
+${grn}# PASS: 300${std}
+${blu}# SKIP: 300${std}
+${lgn}# XFAIL: 300${std}
+# FAIL: 0
+# XPASS: 0
+# ERROR: 0
+$success_footer
+END
+
+ # 1 failure and lots of non-failures means failure.
+ (cat tap && echo "not ok" && cat tap) > all.test
+ test `wc -l <all.test` -eq 601 || Exit 99 # Sanity check.
+ echo 1..601 >> all.test # Test plan.
+ do_check --fail <<END
+$failure_header
+${brg}# TOTAL: 601${std}
+${grn}# PASS: 200${std}
+${blu}# SKIP: 200${std}
+${lgn}# XFAIL: 200${std}
+${red}# FAIL: 1${std}
+# XPASS: 0
+# ERROR: 0
+$failure_footer
+END
+
+ # 1 error and lots of non-failures means failure.
+ (cat tap && sed 30q tap && echo 'Bail out!') > all.test
+ test `wc -l <all.test` -eq 331 || Exit 99 # Sanity check.
+ echo 1..331 >> all.test # Test plan.
+ do_check --fail <<END
+$failure_header
+${brg}# TOTAL: 331${std}
+${grn}# PASS: 110${std}
+${blu}# SKIP: 110${std}
+${lgn}# XFAIL: 110${std}
+# FAIL: 0
+# XPASS: 0
+${mgn}# ERROR: 1${std}
+$failure_footer
+END
+
+done # use_colors in "yes" "no"
+
+:
diff --git a/tests/tap-todo-skip-together.test
b/tests/tap-todo-skip-together.test
new file mode 100755
index 0000000..1528ca3
--- /dev/null
+++ b/tests/tap-todo-skip-together.test
@@ -0,0 +1,57 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support:
+# - TODO and SKIP directives on the same line: the first one wins
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TEST_LOG_COMPILER = cat
+TESTS = all.test
+END
+
+cat > all.test <<END
+1..2
+ok 1 # SKIP TODO
+not ok 2 # TODO SKIP
+END
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+$MAKE check >stdout || { cat stdout; Exit 1; }
+cat stdout
+
+count_test_results total=2 pass=0 fail=0 xpass=0 xfail=1 skip=1 error=0
+
+grep '^SKIP: all\.test 1 ' stdout
+grep '^XFAIL: all\.test 2 ' stdout
+
+:
diff --git a/tests/tap-todo-skip.test b/tests/tap-todo-skip.test
new file mode 100755
index 0000000..f82da7b
--- /dev/null
+++ b/tests/tap-todo-skip.test
@@ -0,0 +1,177 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support:
+# - TODO and SKIP directives are case-insensitive
+# - TODO and SKIP directives can be followed optionally by a colon ":"
+# and by an optional explanation.
+# - our driver isn't fooled into recognizing TODO and SKIP directives
+# spuriously
+# - the reasons for TODO and SKIP, if present, are nicely printed in
+# the testsuite progress output
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TEST_LOG_COMPILER = cat
+TESTS = all.test
+END
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+# ----------------------------------------------------- #
+# Check all possible combinations of: #
+# - uppercase/lowercase #
+# - with/without colon character ":" #
+# - with/without explanatory message #
+# in TODO and SKIP directives. #
+# ----------------------------------------------------- #
+
+# There are 2 * 2^6 + 2 * 2^6 = 256 tests.
+echo 1..256 > all.test
+
+# These nested loops are clearer without indentation.
+
+for c1 in t T; do
+for c2 in o O; do
+for c3 in d D; do
+for c4 in o O; do
+for ex in '' ':' ' foo' ': foo'; do
+ echo "not ok # $c1$c2$c3$c4$ex"
+ echo "not ok# $c1$c2$c3$c4$ex"
+done; done; done; done; done >> all.test
+
+for c1 in s S; do
+for c2 in k K; do
+for c3 in i I; do
+for c4 in p P; do
+for ex in '' ':' ' foo' ': foo'; do
+ echo "ok # $c1$c2$c3$c4$ex"
+ echo "ok# $c1$c2$c3$c4$ex"
+done; done; done; done; done >> all.test
+
+cat all.test # For debugging.
+
+$MAKE check >stdout || { cat stdout; Exit 1; }
+cat stdout
+
+count_test_results total=256 pass=0 fail=0 xpass=0 xfail=128 skip=128 error=0
+
+# -------------------------------------------------------- #
+# TODO ans SKIP directives aren't recognized spuriously. #
+# -------------------------------------------------------- #
+
+cat > all.test <<'END'
+1..9
+ok TODO
+ok - TODO
+ok 3 TODO
+ok 4 - TODO
+ok SKIP
+ok - SKIP
+ok 7 SKIP
+ok 8 - SKIP
+ok 9
+END
+
+$MAKE check >stdout || { cat stdout; Exit 1; }
+cat stdout
+
+count_test_results total=9 pass=9 fail=0 xpass=0 xfail=0 skip=0 error=0
+
+# --------------------------------------------------------------------- #
+# Some more ad-hoc checks, also verifying that the testsuite progress #
+# output on console behaves as expected. #
+# --------------------------------------------------------------------- #
+
+cat > stub <<END
+1 # TODO
+2 # TODO foo?
+3 # TODO: bar!
+4 aa # TODO
+5 bb # TODO fnord 5
+6 cc # TODO:${tab}fnord 6
+7 - x # TODO
+8 - y # TODO fnord $tab 8
+9 - z # TODO: fnord 9 $tab
+10# TODO 0
+11$tab# TODO 1
+12 $tab$tab # TODO 2
+13 asd# TODO 3
+14 sad$tab# TODO 4
+15 das$tab$tab # TODO 5
+END
+
+# For "TODO" directives leading to XPASS results.
+
+sed -e 's/^/ok /' stub > all.test
+sed -e 's/^/XPASS: all.test /' stub > exp
+
+cat all.test
+$MAKE check >stdout && { cat stdout; Exit 1; }
+cat stdout
+count_test_results total=15 pass=0 fail=0 xpass=15 xfail=0 skip=0 error=0
+
+LC_ALL=C grep '^[A-Z][A-Z]*:' stdout > got
+cat exp
+cat got
+#diff exp got #XXX
+
+# For "TODO" directives leading to XFAIL results.
+
+sed -e 's/^/not ok /' stub > all.test
+sed -e 's/^/XFAIL: all.test /' stub > exp
+
+cat all.test
+$MAKE check >stdout || { cat stdout; Exit 1; }
+cat stdout
+count_test_results total=15 pass=0 fail=0 xpass=0 xfail=15 skip=0 error=0
+
+LC_ALL=C grep '^[A-Z][A-Z]*:' stdout > got
+cat exp
+cat got
+#diff exp got # XXX
+
+# For "SKIP" directives.
+
+sed -e 's/^/ok /' -e 's/TODO/SKIP/' stub > all.test
+sed 's/^/SKIP: all.test /' stub > exp
+
+cat all.test
+$MAKE check >stdout || { cat stdout; Exit 1; }
+cat stdout
+count_test_results total=15 pass=0 fail=0 xpass=0 xfail=0 skip=15 error=0
+
+LC_ALL=C grep '^[A-Z][A-Z]*:' stdout > got
+cat exp
+cat got
+#diff exp got # XXX
+
+:
diff --git a/tests/tap-whitespace-normalization.test
b/tests/tap-whitespace-normalization.test
new file mode 100755
index 0000000..4a99e15
--- /dev/null
+++ b/tests/tap-whitespace-normalization.test
@@ -0,0 +1,179 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support: whitespace normalization (or lack thereof) in the testsuite
+# progress output on console. We keep all these checks here in a single
+# script so that a potential cosmetic change in the output format won't
+# force us to tweak dozens of other tests (hopefully).
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TEST_LOG_COMPILER = cat
+TESTS =
+END
+
+: > exp
+
+# Just to be sure.
+sp=' '
+tab=' '
+
+#-----------------------------------------------------------------------
+
+cat > numbers.test <<END
+1..20
+ok${tab}1
+ok${sp}${sp}2
+ok${sp}${sp}${tab}${sp}${tab}${tab}3
+ok 4${sp}
+ok 5${tab}
+ok 6${sp}${sp}
+ok ${tab} ${tab}${tab}3
+END
+
+#-----------------------------------------------------------------------
+
+# "Bail out!" magic.
+
+echo TESTS += bailout.test >> Makefile.am
+
+cat > bailout.test <<END
+1..1
+Bail out!${tab} ${tab}We're out of disk space.
+ok 1
+END
+
+cat >> exp <<END
+ERROR: bailout.test - Bail out! We're out of disk space.
+END
+
+echo TESTS += bailout2.test >> Makefile.am
+
+cat > bailout2.test <<END
+1..1
+Bail out!foo${tab}${sp}
+ok 1
+END
+
+cat >> exp <<END
+ERROR: bailout2.test - Bail out! foo
+END
+
+#-----------------------------------------------------------------------
+
+# Diagnostic lines.
+
+echo AM_TEST_LOG_DRIVER_FLAGS = --comments >> Makefile.am
+echo TESTS += cmnt.test >> Makefile.am
+
+cat > cmnt.test <<END
+1..1
+ok 1
+#Leading whitespace gets added
+# ${tab}${tab} ${tab}Extra leading whitespace is stripped
+# Trailing whitespace is stripped ${tab} ${tab}${tab}
+# Middle${tab}whitespace is${tab} ${tab}${tab} kept
+# ${tab} And only${tab}middle ${tab}whitespace ${tab}${tab} ${tab}
+END
+
+cat >> exp <<END
+PASS: cmnt.test 1
+# cmnt.test: Leading whitespace gets added
+# cmnt.test: Extra leading whitespace is stripped
+# cmnt.test: Trailing whitespace is stripped
+# cmnt.test: Middle${tab}whitespace is${tab} ${tab}${tab} kept
+# cmnt.test: And only${tab}middle ${tab}whitespace
+END
+
+#-----------------------------------------------------------------------
+
+# Skip whole test (from TAP plan).
+
+echo TESTS += skipall.test >> Makefile.am
+echo TESTS += skipall2.test >> Makefile.am
+
+cat > skipall.test <<END
+1..0${tab}${tab} #${tab} ${tab}Strip trailing & leading ${tab}${tab}
+ok 1
+END
+
+cat > mu.test <<END
+1..0 #foo $weirdchars
+# TAP errors in here shouldn't be seen, as the test is skipped.
+1..2
+ok 1
+ok 22
+not ok 333
+1..7
+END
+
+chmod a+x all.test
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+$MAKE check >stdout || { cat stdout; Exit 1; }
+cat stdout
+
+count_test_results total=4 pass=0 fail=0 xpass=0 xfail=0 skip=4 error=0
+
+grep '^SKIP: foo\.test$' stdout
+grep '^SKIP: bar\.test$' stdout
+grep '^SKIP: wget\.test - wget(1) not installed$' stdout
+grep "^SKIP: curl\\.test #${tab} ${tab}Can't connect to gnu\\.org!
${tab}${tab}$" stdout
+grep "^SKIP: mu\\.test - foo $weirdchars$" stdout
+test `grep -c ': .*\.test' stdout` -eq 5
+
+#-----------------------------------------------------------------------
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+# Don't care about exit status or number of test results, they should be
+# checked for in many other tests.
+$MAKE check >stdout || :
+cat stdout
+
+LC_ALL=C sort exp > t
+mv -f t exp
+
+# We need the sort below to account for parallel make usage.
+LC_ALL=C grep '[a-z0-9][a-z0-9]*\.test' stdout | LC_ALL=C sort > got
+
+cat exp
+cat got
+diff exp got
+
+#Exit 1 # TODO finish it
+
+:
diff --git a/tests/tap-with-and-without-number.test
b/tests/tap-with-and-without-number.test
new file mode 100755
index 0000000..fac10dc
--- /dev/null
+++ b/tests/tap-with-and-without-number.test
@@ -0,0 +1,58 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support:
+# - numbered and unnumbered test results can coexist
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TEST_LOG_COMPILER = cat
+TESTS = all.test
+END
+
+cat > all.test <<'END'
+1..7
+ok 1 foo
+ok 2 - foo2
+ok - bar
+not ok baz # TODO
+not ok 4 - asd # TODO
+ok 5 # SKIP
+ok zardoz
+END
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+$MAKE check >stdout || { cat stdout; Exit 1; }
+cat stdout
+count_test_results total=7 pass=4 fail=0 xpass=0 xfail=2 skip=1 error=0
+
+:
diff --git a/tests/tap-xfail-tests.test b/tests/tap-xfail-tests.test
new file mode 100755
index 0000000..82a39e9
--- /dev/null
+++ b/tests/tap-xfail-tests.test
@@ -0,0 +1,87 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP and $(XFAIL_TESTS): test results without directives are turned from
+# PASS to XPASS and from FAIL to XFAIL; other results are unchanged.
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TEST_LOG_COMPILER = cat
+TESTS = the.test
+XFAIL_TESTS = $(TESTS)
+END
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+cat > the.test <<END
+1..6
+ok 1
+not ok 2
+ok 3 # TODO
+not ok 4 # TODO
+ok 5 # SKIP
+# The next should count as a failure (and thus will be
+# rendered into an expected failure by XFAIL_TESTS).
+not ok 6 # SKIP
+Bail out!
+END
+
+$MAKE check >stdout && { cat stdout; Exit 1; }
+cat stdout
+
+count_test_results total=7 pass=0 fail=0 xpass=2 xfail=3 skip=1 error=1
+
+grep '^XPASS: the\.test 1$' stdout
+grep '^XFAIL: the\.test 2$' stdout
+grep '^XPASS: the\.test 3 # TODO' stdout
+grep '^XFAIL: the\.test 4 # TODO' stdout
+grep '^SKIP: the\.test 5 # SKIP' stdout
+grep '^XFAIL: the\.test 6 # SKIP' stdout
+grep '^ERROR: the\.test - Bail out!' stdout
+
+test `$FGREP -c ': the.test' stdout` -eq 7
+
+# Check that the exit status of the testsuite is indeed 0 when we
+# would expect success.
+
+cat > the.test <<END
+1..3
+not ok 1
+ok 2 # SKIP
+not ok 3 # TODO
+END
+
+$MAKE check >stdout || { cat stdout; Exit 1; }
+cat stdout
+
+count_test_results total=3 pass=0 fail=0 xpass=0 xfail=2 skip=1 error=0
+
+:
diff --git a/tests/test-driver-custom-multitest-recheck.test
b/tests/test-driver-custom-multitest-recheck.test
index 137ff96..8bbcc7c 100755
--- a/tests/test-driver-custom-multitest-recheck.test
+++ b/tests/test-driver-custom-multitest-recheck.test
@@ -20,6 +20,7 @@
# support, but also serves as a "usability test" for our APIs.
# See also related tests `test-driver-custom-multitest-recheck2.test'
# and `parallel-tests-recheck-override.test'.
+# Keep in sync with 'tap-recheck.test'.
parallel_tests=yes
. ./defs || Exit 1
--
1.7.2.3