[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [PATCH 1/2] test harness: allow more metadata in log files
|
From: |
Stefano Lattarini |
|
Subject: |
Re: [PATCH 1/2] test harness: allow more metadata in log files |
|
Date: |
Sun, 31 Jul 2011 22:43:47 +0200 |
|
User-agent: |
KMail/1.13.3 (Linux/2.6.30-2-686; KDE/4.4.4; i686; ; ) |
On Friday 29 July 2011, Stefano Lattarini wrote:
> Following up on this:
> <https://lists.gnu.org/archive/html/automake-patches/2011-07/msg00070.html>
>
> On Tuesday 19 July 2011, Stefano Lattarini wrote:
> > On Monday 18 July 2011, Ralf Wildenhues wrote:
> > > * Stefano Lattarini wrote on Fri, Jul 15, 2011 at 12:36:17AM CEST:
> > > > The new code for parsing the testsuite-generated `.log' files,
> > > > as introduced in commit `v1.11-872-gc96b881', considers each
> > > > `:test-result:' field anywhere in a `.log' file as a declaration
> > > > of a test result, and accounts for it as such in the testsuite
> > > > summary. Unfortunately this could easily cause spurious test
> > > > failures being reported in the testsuite summary. This happened
> > > > in practice with the Automake's own testsuite; for example:
> > > >
> > > > $ make check TESTS='check12-p.test'; echo exit: $?
> > > > ...
> > > > PASS: check12-p.test
> > > > =====================================
> > > > 4 of 5 tests failed
> > > > See tests/test-suite.log
> > > > Please report to address@hidden
> > > > =====================================
> > > > make[2]: *** [test-suite.log] Error 1
> > > > make: *** [check-am] Error 2
> > > > exit: 2
> > > >
> > > > This change introduces a new special `:test-result:' "END", that,
> > > > when seen, prevents the rest of the log file from being parsed.
> > > >
> > > > For more information, refer to the thread:
> > > > <http://lists.gnu.org/archive/html/automake-patches/2011-06/msg00199.html>
> > > >
> > > > * lib/am/check.am ($(TEST_SUITE_LOG)): Stop the parsing of a log
> > > > file as soon as the special ":test-result:END" directive is seen.
> > > > Related changes and enhancements.
> > > > * lib/test-driver: Protect the rest of the log after the result
> > > > lined with a ":test-result:END" directive.
> > > > * tests/parallel-tests-no-spurious-summary.test: New test.
> > > > * tests/test-driver-end-test-results.test: Likewise.
> > > > * tests/Makefile.am (TESTS): Update.
> > >
> > > I'm still not sold on this. It is not as robust as a test protocol
> > > could be; also I haven't seen this approach being used in any other test
> > > suite environments. Whether some line is considered having a result or
> > > not depends on (possibly far-away) context, on whether aggregation with
> > > other results has happened. When passed through email, misquoting can
> > > change not only the interpretation of the misquoted text (which could be
> > > expected) but possibly also of later, correctly quoted (or not quoted)
> > > text.
> > >
> > Good points. Hmm... the more I think about this issue, the more I'm
> > convinced that we are (ok, I am ;-) starting to unduly use the `.log'
> > files for two ortoghonal purposes:
> > 1. keeping a verbose and detailed log of a test script run; and
> > 2. keeping a sort of "dumb database" of test results, which is used
> > to determine the final testsuite summary (and also by "make recheck"
> > to decide which tests to re-run).
> > Maybe it's time we begin to use two different set of files for this two
> > different roles? E.g., for each `foo.test', we generate a `foo.log' file
> > and a, say, `foo.tdat' (Test DATa file) containing the test results and
> > possibly other metadata used by our harness? This might also allow for
> > easier extensibility of such metadata in the future.
> >
> I'm pursuing this strategy now; it requires some trickery, but it seems
> to work quite well, and not to cause any codebase bloating. Also, the
> documentation of the new behaviour will be clearer and smaller, and the
> addition of new complexity in the testsuite harness will be paid off with
> a decrease in complexity of the test drivers (a win in the long term, as
> we plan to add more test drivers in the future).
>
> Here is a refactoring/preparatory patch, which makes the metadata in the
> log files more flexible and more sophisticated; while doing so, it IMHO
> also shows that a new set of generated files holding only test metadata
> would be more natural and flexible.
>
> Given what I've said, please do not judge this patch in isolation;
> consider it only a "transitional step" towards the new APIs.
>
And here is the patch implementing this new APIs. It still lacks
documentation in `doc/automake.texi' (I hope I'll be able to post that
tomorrow), and have various rough edges and a couple of speed issues,
but IMHO is well-comment, well tested, and clear in its intents.
In case you ask why have I chosen the `.trs' suffix for files containing
the test results and related metadata: it stands for Test ReSults, and
it seems not to be already used for other significant purposes; see:
- http://filext.com/file-extension/trs
- http://www.file-extensions.org/search/?searchstring=trs
Regards,
Stefano
From 8488bda1e71e0f5ef4b69544576eb4c64a8ae9fa Mon Sep 17 00:00:00 2001
Message-Id: <address@hidden>
From: Stefano Lattarini <address@hidden>
Date: Thu, 28 Jul 2011 12:25:22 +0200
Subject: [PATCH] test harness: new `.trs' files to hold metadata of run tests
---
ChangeLog | 83 ++++++++++
automake.in | 4 +
lib/Automake/tests/Makefile.in | 118 ++++++++------
lib/am/check.am | 155 ++++++++++--------
lib/am/check2.am | 16 +-
lib/tap-driver | 42 +++---
lib/test-driver | 28 ++--
tests/.gitignore | 1 +
tests/Makefile.am | 16 ++-
tests/Makefile.in | 134 +++++++++-------
tests/parallel-tests-make-n.test | 107 ++++++++++++
...nreadable-log.test => parallel-tests-once.test} | 32 ++---
...ble-log.test => parallel-tests-unreadable.test} | 45 ++++--
tests/parallel-tests9.test | 9 -
tests/test-driver-create-log-dir.test | 18 ++-
tests/test-driver-custom-multitest.test | 1 +
tests/test-driver-custom-no-html.test | 4 +-
tests/test-driver-custom-xfail-tests.test | 36 +++--
tests/test-driver-custom.test | 4 +
tests/test-driver-end-metadata.test | 120 --------------
tests/test-driver-metadata-no-leading-space.test | 94 -----------
tests/test-driver-strip-vpath.test | 2 +
...test => test-driver-trs-suffix-registered.test} | 62 ++++----
...obal-log.test => test-metadata-global-log.test} | 132 +++++++++-------
...ver-recheck.test => test-metadata-recheck.test} | 22 ++-
tests/test-metadata-results.test | 169 ++++++++++++++++++++
tests/test-missing.test | 72 +++++++++
...ests-unreadable-log.test => test-missing2.test} | 44 +++---
tests/test-trs-basic.test | 138 ++++++++++++++++
tests/test-trs-recover.test | 166 +++++++++++++++++++
tests/test-trs-recover2.test | 133 +++++++++++++++
tests/trivial-test-driver | 58 +++-----
32 files changed, 1392 insertions(+), 673 deletions(-)
create mode 100755 tests/parallel-tests-make-n.test
copy tests/{parallel-tests-unreadable-log.test => parallel-tests-once.test}
(59%)
copy tests/{parallel-tests-unreadable-log.test =>
parallel-tests-unreadable.test} (58%)
delete mode 100755 tests/test-driver-end-metadata.test
delete mode 100755 tests/test-driver-metadata-no-leading-space.test
copy tests/{parallel-tests-unreadable-log.test =>
test-driver-trs-suffix-registered.test} (51%)
rename tests/{test-driver-global-log.test => test-metadata-global-log.test}
(62%)
rename tests/{test-driver-recheck.test => test-metadata-recheck.test} (93%)
create mode 100755 tests/test-metadata-results.test
create mode 100755 tests/test-missing.test
rename tests/{parallel-tests-unreadable-log.test => test-missing2.test} (56%)
create mode 100755 tests/test-trs-basic.test
create mode 100755 tests/test-trs-recover.test
create mode 100755 tests/test-trs-recover2.test
diff --git a/ChangeLog b/ChangeLog
index f40b307..82a8981 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,86 @@
+2011-07-31 Stefano Lattarini <address@hidden>
+
+ test harness: use new `.trs' files to hold test metadata
+ With this change, the test harness will keep test metadata in
+ dedicated `.trs' files, instead of having them embedded into the
+ `.log' files. This allows for easier forward-compatibility and
+ extension of test metadata, and for more flexibility in the
+ format of the `.log' files. Note that this change makes the
+ `:end-metadata:' field obsolete.
+ * automake.in (handle_tests): When bringing in the content of
+ `check2.am', substitute %BASE% with the basename of the `.log'
+ file being created by a rule. Add the generated `.trs' files
+ to the list of files to be cleaned by "make mostlyclean".
+ * lib/am/check.am (am__test_driver_flags): Rename ...
+ (am__common_driver_flags): ... to this, and remove the flags
+ `--test-name' and `--log-file' from it: they are now define in
+ the proper rules in `check2.am'.
+ (am__TEST_BASES): New internal variable, holding the names of
+ the tests, with any registered extension removed.
+ (am__stealth_MAKE): New internal variable, can be used instead of
+ $(MAKE) in recipes requiring a recursive call to make, but which
+ are not intended to be executed by "make -n".
+ (.log.trs): New suffix rule, to recover from deletion of `.trs'
+ files.
+ ($(TEST_SUITE_LOG)): Almost completely rewritten to follow the
+ new API of "test logs in `.log' files, test metadata in `.trs'
+ files". It goes to some length to work correctly in face of
+ unreadable or missing `.log' and `.trs' files, and to error out
+ with proper error messages when this is not possible.
+ [%?PARALLEL_TESTS%] (check-TESTS): Also remove relevant "stale"
+ `.trs' files (in addition to `.log files) before remaking the
+ $(TEST_SUITE_LOG).
+ (recheck, recheck-html): Look for the `:recheck:' field in the
+ `.trs' files, not in the `.log' files.
+ * lib/am/check2.am (?GENERIC?%EXT%.log, ?!GENERIC?%OBJ%): Adjust
+ the call to the test driver, in particularly passing the new
+ option `--result-file', which specifies the path for the `.trs'
+ file where the metadata for the test run will be saved.
+ [%am__EXEEXT%] (?GENERIC?%EXT%$(EXEEXT).log): Likewise.
+ * lib/tap-driver ($USAGE): Adjust the help screen.
+ (Getopt::Long::GetOptions): Handle the `--result-file' option,
+ through the use of ...
+ ($result_file): ... this new global variable.
+ (finish): Write metadata for the test run to `$result_file' rather
+ then to `$log_file', through the use of ...
+ (write_test_results): ... this new function.
+ * lib/test-driver (print_usage): Update the help screen.
+ (Option parsing): Handle the `--result-file' option, through the
+ use of ...
+ ($resfile): ... this new global variable.
+ (Main code): Write metadata for the test run to `$resfile' rather
+ than to `$logfile'.
+ Minor related adjustments to comments.
+ * tests/.gitignore: Ignore `*.trs' files.
+ * tests/parallel-tests-unreadable-log.test: Moved ...
+ * tests/parallel-tests-unreadable.test: ... to this, and extended
+ to also check the semantics for unreadable `.trs' files.
+ * tests/test-driver-end-metadata.test: Deleted as obsolete.
+ * tests/test-driver-metadata-no-leading-space.test: Likewise.
+ * tests/test-driver-global-log.test: Renamed ...
+ * tests/test-metadata-global-log.test: ... to this, and modified
+ as to verify the new APIs and semantics.
+ * tests/test-driver-recheck: Renamed ...
+ * tests/test-metadata-recheck.test: ... to this, and modified
+ likewise.
+ * tests/parallel-tests-once.test: New test.
+ * tests/parallel-tests-make-n.test: Likewise.
+ * test-metadata-results.test: Likewise.
+ * test-missing.test: Likewise.
+ * test-missing2.test: Likewise.
+ * test-trs-basic.test: Likewise.
+ * test-trs-recover.test: Likewise.
+ * test-trs-recover2.test: Likewise.
+ * tests/Makefile.am (TESTS): Update.
+
+## We choose the `.trs' extension (as in Test ReSults) for the files holding
+## our test results. These seems a good choice because such an extension is
+## not already used for other significant purposes; see:
+## - http://filext.com/file-extension/trs
+## - http://www.file-extensions.org/search/?searchstring=trs
+am__TEST_RESULTS = $(am__TEST_BASES:=.trs)
+
+
2011-07-27 Stefano Lattarini <address@hidden>
test harness: allow more metadata in log files
diff --git a/automake.in b/automake.in
index 2ad8dc2..ad58f06 100644
--- a/automake.in
+++ b/automake.in
@@ -5033,6 +5033,7 @@ sub handle_tests
return substr ($obj, 0, length ($obj) - length
($test_suffix)) . '.log'
if substr ($obj, - length ($test_suffix)) eq $test_suffix;
}
+ my $base = $obj;
$obj .= '.log';
# The "test driver" program, deputed to handle tests protocol
used by
# test scripts. By default, it's assumed that no protocol is
used,
@@ -5054,6 +5055,7 @@ sub handle_tests
$output_rules .= file_contents ('check2', new
Automake::Location,
GENERIC => 0,
OBJ => $obj,
+ BASE => $base,
SOURCE => $val,
DRIVER => $driver,
DRIVER_FLAGS => $driver_flags,
@@ -5107,6 +5109,7 @@ sub handle_tests
$output_rules .= file_contents ('check2', new
Automake::Location,
GENERIC => 1,
OBJ => '',
+ BASE => '$*',
SOURCE => '$<',
DRIVER => $driver,
DRIVER_FLAGS => $driver_flags,
@@ -5122,6 +5125,7 @@ sub handle_tests
$clean_files{'$(TEST_LOGS_TMP)'} = MOSTLY_CLEAN;
$clean_files{'$(TEST_LOGS)'} = MOSTLY_CLEAN;
+ $clean_files{'$(am__TEST_BASES:=.trs)'} = MOSTLY_CLEAN;
$clean_files{'$(TEST_SUITE_LOG)'} = MOSTLY_CLEAN;
$clean_files{'$(TEST_SUITE_HTML)'} = MOSTLY_CLEAN;
}
diff --git a/lib/Automake/tests/Makefile.in b/lib/Automake/tests/Makefile.in
index 9b45aa2..d6a6152 100644
--- a/lib/Automake/tests/Makefile.in
+++ b/lib/Automake/tests/Makefile.in
@@ -119,10 +119,8 @@ am__rst_title = sed 's/.*/ &
/;h;s/./=/g;p;x;p;g;p;s/.*//'
# pass "-e" to $(SHELL), and POSIX 2008 even requires this. Work around it
# by disabling -e (using the XSI extension "set +e") if it's set.
am__sh_e_setup = case $$- in *e*) set +e;; esac
-# Default flags passed to all log compiler wrappers.
-am__test_driver_flags = \
- --test-name "$$f" \
- --log-file '$@' \
+# Default flags passed to test drivers.
+am__common_driver_flags = \
--color-tests "$$am__color_tests" \
--enable-hard-errors "$$am__enable_hard_errors" \
--expect-failure "$$am__expect_failure"
@@ -156,6 +154,14 @@ case " $(XFAIL_TESTS) " in \
am__expect_failure=no;; \
esac; \
$(AM_TESTS_ENVIRONMENT) $(TESTS_ENVIRONMENT)
+# The names of the tests, with any registered extension removed. Or
+# equivalently, the names of the test logs, with the `.log' extension
+# renoved. This honours runtime overriding of TESTS and TEST_LOGS.
+am__TEST_BASES = $(TEST_LOGS:.log=)
+# This can be used instead of $(MAKE) in recipes requiring a recursive call
+# to make, but which are not intended to be executed by "make -n". See the
+# GNU make manual for more details.
+am__stealth_MAKE = $(MAKE)
RECHECK_LOGS = $(TEST_LOGS)
AM_RECURSIVE_TARGETS = check check-html recheck recheck-html
TEST_SUITE_LOG = test-suite.log
@@ -289,7 +295,7 @@ EXTRA_DIST = $(TESTS)
all: all-am
.SUFFIXES:
-.SUFFIXES: .html .log .pl
+.SUFFIXES: .html .log .pl .trs
$(srcdir)/Makefile.in: $(srcdir)/Makefile.am $(am__configure_deps)
@for dep in $?; do \
case '$(am__configure_deps)' in \
@@ -329,43 +335,46 @@ CTAGS:
cscope cscopelist:
+# Recover from deleted `.trs' file; this should ensure that
+# "rm -f foo.log; make foo.trs" re-run `foo.test', and re-create
+# both `foo.log' and `foo.trs'.
+.log.trs:
+ rm -f $< $@ && $(am__stealth_MAKE) $(AM_MAKEFLAGS) $<
+
$(TEST_SUITE_LOG): $(TEST_LOGS)
@$(am__sh_e_setup); $(am__tty_colors); \
- tab=' '; \
- nlinit=`echo 'nl="'; echo '"'`; eval "$$nlinit"; unset nlinit; \
- IFS=" $$tab$$nl"; oIFS=$$IFS; \
- ws="[ $$tab]"; \
- list='$(TEST_LOGS)'; \
- list2=`for f in $$list; do test ! -r $$f || echo $$f; done`; \
- results1=`for f in $$list; do test -r $$f || echo ERROR; done`; \
- results2=''; \
- exec 5<&0; \
- for f in $$list2; do \
- exec 0<$$f; \
- IFS=''; \
- while read line; do \
- IFS=$$oIFS; \
- case $$line in \
- :end-metadata:*) \
- break;; \
- :test-result:*) \
- res=`expr "x$$line" : "x:test-result:$$ws*\\\\(.*\\\\)$$"`; \
- results2="$$results2$$nl$$res";; \
- esac; \
- IFS=''; \
+ bases='$(am__TEST_BASES)'; \
+ ws='[ ]'; \
+ redo_bases=`for b in $$bases; do \
+ test -f $$b.trs && test -r $$b.trs \
+ && test -f $$b.log && test -r $$b.log \
+ || echo $$b; \
+ done`; \
+ if test -n "$$redo_bases"; then \
+ redo_logs=`for i in $$redo_bases; do echo $$i.log; done`; \
+ redo_results=`for i in $$redo_bases; do echo $$i.trs; done`; \
+ rm -f $$redo_logs && rm -f $$redo_results \
+ && $(am__stealth_MAKE) $(AM_MAKEFLAGS) $$redo_logs; \
+ fi; \
+ st=0; \
+ for b in $$redo_bases; do \
+ for e in trs log; do \
+ if test ! -f $$b.$$e || test ! -r $$b.$$e; then \
+ echo "fatal: making $@: failed to create $$b.$$e" >&2; \
+ st=1; \
+ fi; \
done; \
- IFS=$$oIFS; \
- exec 0<&5; \
done; \
- exec 5<&-; \
- results=`echo "$$results1" && echo "$$results2"`; \
- all=`echo "$$results" | sed '/^$$/d' | wc -l | sed -e 's/^[
]*//'`; \
- fail=`echo "$$results" | grep -c '^FAIL'`; \
- pass=`echo "$$results" | grep -c '^PASS'`; \
- skip=`echo "$$results" | grep -c '^SKIP'`; \
- xfail=`echo "$$results" | grep -c '^XFAIL'`; \
- xpass=`echo "$$results" | grep -c '^XPASS'`; \
- error=`echo "$$results" | grep -c '^ERROR'`; \
+ test $$st -eq 0 || exit 1; \
+ results=`for b in $$bases; do echo $$b.trs; done`; \
+ test -n "$$results" || results=/dev/null; \
+ all=` grep "^$$ws*:test-result:" $$results | wc -l`; \
+ pass=` grep "^$$ws*:test-result:$$ws*PASS" $$results | wc -l`; \
+ fail=` grep "^$$ws*:test-result:$$ws*FAIL" $$results | wc -l`; \
+ skip=` grep "^$$ws*:test-result:$$ws*SKIP" $$results | wc -l`; \
+ xfail=`grep "^$$ws*:test-result:$$ws*XFAIL" $$results | wc -l`; \
+ xpass=`grep "^$$ws*:test-result:$$ws*XPASS" $$results | wc -l`; \
+ error=`grep "^$$ws*:test-result:$$ws*ERROR" $$results | wc -l`; \
if test `expr $$fail + $$xpass + $$error` -eq 0; then \
success=true; \
else \
@@ -407,10 +416,15 @@ $(TEST_SUITE_LOG): $(TEST_LOGS)
echo; \
echo ".. contents:: :depth: 2"; \
echo; \
- for f in $$list; do \
- sed -e '/^:end-metadata:/,$$d' $$f | \
- grep "^:copy-in-global-log:$$ws*no$$ws*$$" >/dev/null \
- || { echo && cat $$f; } \
+ for b in $$bases; do \
+ if grep "^$$ws*:copy-in-global-log:$$ws*no$$ws*$$" $$b.trs \
+ >/dev/null; then :; \
+ elif test ! -r $$b.log; then \
+ echo "ERROR: cannot read $$b.log" >&2; \
+ echo && echo "WARNING: could not read $$b.log!"; \
+ else \
+ echo && cat $$b.log; \
+ fi; \
done; \
} >$(TEST_SUITE_LOG).tmp; \
mv $(TEST_SUITE_LOG).tmp $(TEST_SUITE_LOG); \
@@ -437,6 +451,7 @@ $(TEST_SUITE_LOG): $(TEST_LOGS)
# Run all the tests.
check-TESTS:
@list='$(RECHECK_LOGS)'; test -z "$$list" || rm -f $$list
+ @list='$(RECHECK_LOGS:.log=.trs)'; test -z "$$list" || rm -f $$list
@test -z "$(TEST_SUITE_LOG)" || rm -f $(TEST_SUITE_LOG)
@list='$(TEST_LOGS)'; \
list=`for f in $$list; do \
@@ -471,19 +486,19 @@ check-html:
recheck recheck-html:
@ws='[ ]'; \
target=`echo $@ | sed 's,^re,,'`; \
- list='$(TEST_LOGS)'; \
- list=`for f in $$list; do \
- test -f $$f || continue; \
- sed -e '/^:end-metadata:/,$$d' $$f | \
- grep "^:recheck:$$ws*no$$ws*$$" >/dev/null \
- || echo $$f; \
+ bases='$(am__TEST_BASES)'; \
+ list=`for b in $$bases; do \
+ test ! -f $$b.trs && test ! -f $$b.log && continue; \
+ grep "^$$ws*:recheck:$$ws*no$$ws*$$" $$b.trs \
+ >/dev/null 2>&1 || echo $$b.log; \
done | tr '\012\015' ' '`; \
list=`echo "$$list" | sed 's/ *$$//'`; \
$(MAKE) $(AM_MAKEFLAGS) $$target AM_MAKEFLAGS='$(AM_MAKEFLAGS)
TEST_LOGS="'"$$list"'"'
.pl.log:
- @p='$<'; $(am__check_pre) \
- $(PL_LOG_DRIVER) $(am__test_driver_flags) $(AM_PL_LOG_DRIVER_FLAGS)
$(PL_LOG_DRIVER_FLAGS) -- \
- $(PL_LOG_COMPILE) "$$tst" $(AM_TESTS_FD_REDIRECT)
+ @p='$<'; $(am__check_pre) $(PL_LOG_DRIVER) --test-name "$$f" \
+ --log-file '$*.log' --result-file '$*.trs' \
+ $(am__common_driver_flags) $(AM_PL_LOG_DRIVER_FLAGS)
$(PL_LOG_DRIVER_FLAGS) -- $(PL_LOG_COMPILE) "$$tst" \
+ $(AM_TESTS_FD_REDIRECT)
distdir: $(DISTFILES)
@srcdirstrip=`echo "$(srcdir)" | sed 's/[].[^$$\\*]/\\\\&/g'`; \
@@ -544,6 +559,7 @@ mostlyclean-generic:
-test -z "$(TEST_LOGS_TMP)" || rm -f $(TEST_LOGS_TMP)
-test -z "$(TEST_SUITE_HTML)" || rm -f $(TEST_SUITE_HTML)
-test -z "$(TEST_SUITE_LOG)" || rm -f $(TEST_SUITE_LOG)
+ -test -z "$(am__TEST_BASES:=.trs)" || rm -f $(am__TEST_BASES:=.trs)
clean-generic:
diff --git a/lib/am/check.am b/lib/am/check.am
index 272fc0a..c406e9f 100644
--- a/lib/am/check.am
+++ b/lib/am/check.am
@@ -83,10 +83,8 @@ am__rst_title = sed 's/.*/ &
/;h;s/./=/g;p;x;p;g;p;s/.*//'
# by disabling -e (using the XSI extension "set +e") if it's set.
am__sh_e_setup = case $$- in *e*) set +e;; esac
-# Default flags passed to all log compiler wrappers.
-am__test_driver_flags = \
- --test-name "$$f" \
- --log-file '$@' \
+# Default flags passed to test drivers.
+am__common_driver_flags = \
--color-tests "$$am__color_tests" \
--enable-hard-errors "$$am__enable_hard_errors" \
--expect-failure "$$am__expect_failure"
@@ -124,64 +122,71 @@ case " $(XFAIL_TESTS) " in
\
esac; \
$(AM_TESTS_ENVIRONMENT) $(TESTS_ENVIRONMENT)
+# The names of the tests, with any registered extension removed. Or
+# equivalently, the names of the test logs, with the `.log' extension
+# renoved. This honours runtime overriding of TESTS and TEST_LOGS.
+am__TEST_BASES = $(TEST_LOGS:.log=)
+
+# This can be used instead of $(MAKE) in recipes requiring a recursive call
+# to make, but which are not intended to be executed by "make -n". See the
+# GNU make manual for more details.
+am__stealth_MAKE = $(MAKE)
+
+# Recover from deleted `.trs' file; this should ensure that
+# "rm -f foo.log; make foo.trs" re-run `foo.test', and re-create
+# both `foo.log' and `foo.trs'.
+.log.trs:
+ rm -f $< $@ && $(am__stealth_MAKE) $(AM_MAKEFLAGS) $<
+
$(TEST_SUITE_LOG): $(TEST_LOGS)
@$(am__sh_e_setup); $(am__tty_colors); \
- tab=' '; \
-## The following line will define the `$nl' shell variable to a literal
-## newline. Idiom taken from the Autoconf manual.
- nlinit=`echo 'nl="'; echo '"'`; eval "$$nlinit"; unset nlinit; \
-## Sanitize internal field separator for the shell: "space tab newline".
- IFS=" $$tab$$nl"; oIFS=$$IFS; \
-## Regular expression to match a whitespace.
- ws="[ $$tab]"; \
-## All test logs.
- list='$(TEST_LOGS)'; \
-## Readable test logs.
- list2=`for f in $$list; do test ! -r $$f || echo $$f; done`; \
-## Each unreadable test log counts as a failed test.
- results1=`for f in $$list; do test -r $$f || echo ERROR; done`; \
-## Now we're going to extract the outcome of all the testcases from the
-## test logs.
- results2=''; \
-## Can't do the redirections directly in the "for" loop below, otherwise
-## some shells would run the loop body in a subshell.
- exec 5<&0; \
- for f in $$list2; do \
- exec 0<$$f; \
-## We want to support special reStructuredText fields only when they're
-## placed at the beginning of a line (as documented in the manual), so
-## we have to temporarily clear the $IFS variable to prevent automatic
-## whitespace stripping by the shell.
- IFS=''; \
- while read line; do \
- IFS=$$oIFS; \
- case $$line in \
-## Special field that tells not to scan the rest of the log file.
- :end-metadata:*) \
- break;; \
-## A test test result -- register it.
- :test-result:*) \
- res=`expr "x$$line" : "x:test-result:$$ws*\\\\(.*\\\\)$$"`; \
- results2="$$results2$$nl$$res";; \
- esac; \
-## For the next loop.
- IFS=''; \
+ bases='$(am__TEST_BASES)'; \
+ ws='[ ]'; \
+## We need to ensures that all the required `.trs' and `.log' files will
+## be present and readable. The direct dependencies of $(TEST_SUITE_LOG)
+## only ensure that all the `.log' files exists; they don't ensure that
+## the `.log' files are readable, and worse, they don't ensure that the
+## `.trs' files even exist.
+ redo_bases=`for b in $$bases; do \
+ test -f $$b.trs && test -r $$b.trs \
+ && test -f $$b.log && test -r $$b.log \
+ || echo $$b; \
+ done`; \
+ if test -n "$$redo_bases"; then \
+## Uh-oh, either some `.log' files were unreadable, or some `.trs' files
+## were missing (or unreadable). We need to re-run the corresponding
+## tests in order to re-create them.
+ redo_logs=`for i in $$redo_bases; do echo $$i.log; done`; \
+ redo_results=`for i in $$redo_bases; do echo $$i.trs; done`; \
+ rm -f $$redo_logs && rm -f $$redo_results \
+ && $(am__stealth_MAKE) $(AM_MAKEFLAGS) $$redo_logs; \
+ fi; \
+## Sanity check: each unreadable or non-existent test result file should
+## has been properly remade at this point, as should the corresponding log
+## file.
+ st=0; \
+ for b in $$redo_bases; do \
+ for e in trs log; do \
+ if test ! -f $$b.$$e || test ! -r $$b.$$e; then \
+ echo "fatal: making $@: failed to create $$b.$$e" >&2; \
+ st=1; \
+ fi; \
done; \
-## Restore original standard input and internal field separator.
- IFS=$$oIFS; \
- exec 0<&5; \
done; \
-## We don't need this anymore.
- exec 5<&-; \
-## Prepare data for the test suite summary.
- results=`echo "$$results1" && echo "$$results2"`; \
- all=`echo "$$results" | sed '/^$$/d' | wc -l | sed -e 's/^[
]*//'`; \
- fail=`echo "$$results" | grep -c '^FAIL'`; \
- pass=`echo "$$results" | grep -c '^PASS'`; \
- skip=`echo "$$results" | grep -c '^SKIP'`; \
- xfail=`echo "$$results" | grep -c '^XFAIL'`; \
- xpass=`echo "$$results" | grep -c '^XPASS'`; \
- error=`echo "$$results" | grep -c '^ERROR'`; \
+ test $$st -eq 0 || exit 1; \
+## List of test result files.
+ results=`for b in $$bases; do echo $$b.trs; done`; \
+ test -n "$$results" || results=/dev/null; \
+## Prepare data for the test suite summary. These do not take into account
+## unreadable test results, but they'll be appropriately updated later if
+## needed.
+ all=` grep "^$$ws*:test-result:" $$results | wc -l`; \
+ pass=` grep "^$$ws*:test-result:$$ws*PASS" $$results | wc -l`; \
+ fail=` grep "^$$ws*:test-result:$$ws*FAIL" $$results | wc -l`; \
+ skip=` grep "^$$ws*:test-result:$$ws*SKIP" $$results | wc -l`; \
+ xfail=`grep "^$$ws*:test-result:$$ws*XFAIL" $$results | wc -l`; \
+ xpass=`grep "^$$ws*:test-result:$$ws*XPASS" $$results | wc -l`; \
+ error=`grep "^$$ws*:test-result:$$ws*ERROR" $$results | wc -l`; \
## Whether the testsuite was successful or not.
if test `expr $$fail + $$xpass + $$error` -eq 0; then \
success=true; \
@@ -233,11 +238,19 @@ $(TEST_SUITE_LOG): $(TEST_LOGS)
echo; \
echo ".. contents:: :depth: 2"; \
echo; \
- for f in $$list; do \
-## FIXME: two forks per test -- this is horrendously inefficient!
- sed -e '/^:end-metadata:/,$$d' $$f | \
- grep "^:copy-in-global-log:$$ws*no$$ws*$$" >/dev/null \
- || { echo && cat $$f; } \
+ for b in $$bases; do \
+## FIXME: one fork per test -- this is horrendously inefficient!
+ if grep "^$$ws*:copy-in-global-log:$$ws*no$$ws*$$" $$b.trs \
+ >/dev/null; then :; \
+## If we cannot read the .log file of a test ...
+ elif test ! -r $$b.log; then \
+## ... print an error message to the console ...
+ echo "ERROR: cannot read $$b.log" >&2; \
+## ... and a warning in the `test-suite.log' file.
+ echo && echo "WARNING: could not read $$b.log!"; \
+ else \
+ echo && cat $$b.log; \
+ fi; \
done; \
} >$(TEST_SUITE_LOG).tmp; \
mv $(TEST_SUITE_LOG).tmp $(TEST_SUITE_LOG); \
@@ -272,6 +285,7 @@ RECHECK_LOGS = $(TEST_LOGS)
check-TESTS:
## Expand $(RECHECK_LOGS) only once, to avoid exceeding line length limits.
@list='$(RECHECK_LOGS)'; test -z "$$list" || rm -f $$list
+ @list='$(RECHECK_LOGS:.log=.trs)'; test -z "$$list" || rm -f $$list
## We always have to remove TEST_SUITE_LOG, to ensure its rule is run
## in any case even in lazy mode: otherwise, if no test needs rerunning,
## or a prior run plus reruns all happen within the same timestamp
@@ -335,13 +349,14 @@ AM_RECURSIVE_TARGETS += check-html
recheck recheck-html:
@ws='[ ]'; \
target=`echo $@ | sed 's,^re,,'`; \
- list='$(TEST_LOGS)'; \
- list=`for f in $$list; do \
- test -f $$f || continue; \
-## FIXME: two forks per test -- this is horrendously inefficient!
- sed -e '/^:end-metadata:/,$$d' $$f | \
- grep "^:recheck:$$ws*no$$ws*$$" >/dev/null \
- || echo $$f; \
+ bases='$(am__TEST_BASES)'; \
+ list=`for b in $$bases; do \
+## Skip tests that haven't been run, but recover gracefully from deleted
+## `.trs' files.
+ test ! -f $$b.trs && test ! -f $$b.log && continue; \
+## FIXME: one fork per test -- this is horrendously inefficient!
+ grep "^$$ws*:recheck:$$ws*no$$ws*$$" $$b.trs \
+ >/dev/null 2>&1 || echo $$b.log; \
done | tr '\012\015' ' '`; \
## This apparently useless munging helps to avoid a nasty bug (a
## segmentation fault!) on Solaris XPG4 make.
diff --git a/lib/am/check2.am b/lib/am/check2.am
index f4fb3c9..8b9a703 100644
--- a/lib/am/check2.am
+++ b/lib/am/check2.am
@@ -14,12 +14,13 @@
## You should have received a copy of the GNU General Public License
## along with this program. If not, see <http://www.gnu.org/licenses/>.
-## From a test file to a log file.
+## From a test file to a .log and .trs file.
?GENERIC?%EXT%.log:
?!GENERIC?%OBJ%: %SOURCE%
- @p='%SOURCE%'; $(am__check_pre) \
- %DRIVER% $(am__test_driver_flags) %DRIVER_FLAGS% -- \
- %COMPILE% "$$tst" $(AM_TESTS_FD_REDIRECT)
+ @p='%SOURCE%'; $(am__check_pre) %DRIVER% --test-name "$$f" \
+ --log-file '%BASE%.log' --result-file '%BASE%.trs' \
+ $(am__common_driver_flags) %DRIVER_FLAGS% -- %COMPILE% "$$tst" \
+ $(AM_TESTS_FD_REDIRECT)
## If no programs are built in this package, then this rule is removed
## at automake time. Otherwise, %am__EXEEXT% expands to a configure time
@@ -27,7 +28,8 @@
## conflict with the previous one.
if %am__EXEEXT%
?GENERIC?%EXT%$(EXEEXT).log:
- @p='%SOURCE%'; $(am__check_pre) \
- %DRIVER% $(am__test_driver_flags) %DRIVER_FLAGS% -- \
- %COMPILE% "$$tst" $(AM_TESTS_FD_REDIRECT)
+ @p='%SOURCE%'; $(am__check_pre) %DRIVER% --test-name "$$f" \
+ --log-file '%BASE%.log' --result-file '%BASE%.trs' \
+ $(am__common_driver_flags) %DRIVER_FLAGS% -- %COMPILE% "$$tst" \
+ $(AM_TESTS_FD_REDIRECT)
endif %am__EXEEXT%
diff --git a/lib/tap-driver b/lib/tap-driver
index 9075d35..7681fb4 100755
--- a/lib/tap-driver
+++ b/lib/tap-driver
@@ -15,7 +15,7 @@ my $ME = "tap-driver";
my $USAGE = <<'END';
Usage:
- tap-driver [--help|--version] --test-name=NAME --log-file=PATH
+ tap-driver --test-name=NAME --log-file=PATH --result-file=PATH
[--expect-failure={yes|no}] [--color-tests={yes|no}]
[--enable-hard-errors={yes|no}] [--merge|--no-merge]
[--ignore-exit] [--comments|--no-comments] [--] TEST-COMMAND
@@ -62,12 +62,14 @@ my %cfg = (
my $test_script_name = undef;
my $log_file = undef;
+my $result_file = undef;
Getopt::Long::GetOptions (
'help' => sub { print $HELP; exit 0; },
'version' => sub { print "$ME $VERSION"; exit 0; },
'test-name=s' => \$test_script_name,
'log-file=s' => \$log_file,
+ 'result-file=s' => \$result_file,
'color-tests=s' => \&bool_opt,
'expect-failure=s' => \&bool_opt,
'enable-hard-errors=s' => \&bool_opt,
@@ -164,6 +166,20 @@ TEST_RESULTS :
}
+sub write_test_results ()
+{
+ open RES, ">", $result_file or die "opening $result_file: $!\n";
+ print RES ":recheck: " .
+ (must_recheck ? "yes" : "no") . "\n";
+ print RES ":copy-in-global-log: " .
+ (copy_in_global_log ? "yes" : "no") . "\n";
+ foreach my $result (get_test_results)
+ {
+ print RES ":test-result: $result\n";
+ }
+ close RES or die "closing $result_file: $!\n";
+}
+
sub start (@)
{
# Redirect stderr and stdout to a temporary log file. Save the
@@ -179,35 +195,15 @@ sub start (@)
sub finish ()
{
- open LOG, ">", $log_file or die "opening $log_file: $!\n";
-
- # Some extra trailing empty lines outputted below are required
- # to support reStructuredText -> HTML conversion.
+ write_test_results;
+ open LOG, ">", $log_file or die "opening $log_file: $!\n";
my $global_result = get_global_test_result;
my $global_result_line = "$global_result: $test_script_name";
print LOG "$global_result_line\n";
print LOG "=" x length ($global_result_line) . "\n";
print LOG "\n";
- print LOG ":recheck: " . (must_recheck ? "yes" : "no") . "\n";
- print LOG "\n";
-
- print LOG ":copy-in-global-log: " .
- (copy_in_global_log ? "yes" : "no") . "\n";
- print LOG "\n";
-
- foreach (get_test_results)
- {
- print LOG ":test-result: $_\n";
- print LOG "\n";
- }
-
- # So that the output from the test script won't be parsed for
- # testsuite metadata.
- print LOG ":end-metadata:\n";
- print LOG "\n";
-
close TMP or die "closing $log_file-t: $!\n";
# FIXME: remove this hack!
my $test_output = `cat $log_file-t && rm -f $log_file-t`;
diff --git a/lib/test-driver b/lib/test-driver
index 45b3afe..ccdf873 100755
--- a/lib/test-driver
+++ b/lib/test-driver
@@ -48,7 +48,7 @@ print_usage ()
{
cat <<END
Usage:
- test-driver [--help|--version] --test-name=NAME --log-file=PATH
+ test-driver --test-name=NAME --log-file=PATH --result-file=PATH
[--expect-failure={yes|no}] [--color-tests={yes|no}]
[--enable-hard-errors={yes|no}] [--] TEST-SCRIPT
The \`--test-name' and \`--log-file' options are mandatory.
@@ -59,9 +59,10 @@ END
rst_section () { sed 'p;s/./=/g;p;g'; }
# TODO: better error handling in option parsing (in particular, ensure
-# TODO: $logfile and $test_name are defined).
+# TODO: $logfile, $resfile and $test_name are defined).
test_name= # Used for reporting.
-logfile= # Where to save the result and output of the test script.
+logfile= # Where to save the output of the test script.
+resfile= # Where to save the result(s) the test script.
expect_failure=no
color_tests=no
enable_hard_errors=yes
@@ -71,6 +72,7 @@ while test $# -gt 0; do
--version) echo "test-driver $scriptversion"; exit $?;;
--test-name) test_name=$2; shift;;
--log-file) logfile=$2; shift;;
+ --result-file) resfile=$2; shift;;
--color-tests) color_tests=$2; shift;;
--expect-failure) expect_failure=$2; shift;;
--enable-hard-errors) enable_hard_errors=$2; shift;;
@@ -118,24 +120,14 @@ esac
# Report outcome to console.
echo "${col}${res}${std}: $test_name"
+# Register the test result, and other relevant metadata.
+echo ":test-result: $res (exit status: $estatus)" > $resfile
+echo ":recheck: $recheck" >> $resfile
+echo ":copy-in-global-log: $gcopy" >> $resfile
+
# Now write log file.
{
echo "$res: $test_name (exit: $estatus)" | rst_section
- # Blank lines required for having the output be valid RST.
- echo
- echo ":test-result: $res (exit status: $estatus)"
- echo
- echo ":recheck: $recheck"
- echo
- echo ":copy-in-global-log: $gcopy"
- echo
- # So that the output from the test script won't be parsed for
- # testsuite metadata.
- echo ":end-metadata:"
- echo
- # Use a reStructuredText transition to better separate the test
- # metadata the test registered output.
- echo ------------
echo
cat $tmpfile
} > $logfile
diff --git a/tests/.gitignore b/tests/.gitignore
index 026e261..0d5a032 100644
--- a/tests/.gitignore
+++ b/tests/.gitignore
@@ -5,6 +5,7 @@ instspc-tests.am
parallel-tests.am
*.dir
*.log
+*.trs
*.log-t
*-p.test
instspc-*-build.test
diff --git a/tests/Makefile.am b/tests/Makefile.am
index 9a0c222..104f70b 100644
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -724,9 +724,10 @@ parallel-tests7.test \
parallel-tests8.test \
parallel-tests9.test \
parallel-tests10.test \
+parallel-tests-make-n.test \
parallel-tests-fd-redirect.test \
parallel-tests-am_tests_environment.test \
-parallel-tests-unreadable-log.test \
+parallel-tests-unreadable.test \
parallel-tests-subdir.test \
parallel-tests-interrupt.test \
parallel-tests-reset-term.test \
@@ -739,12 +740,11 @@ parallel-test-driver-install.test \
parallel-tests-no-spurious-summary.test \
parallel-tests-exit-statuses.test \
parallel-tests-console-output.test \
+parallel-tests-once.test \
testsuite-summary-color.test \
testsuite-summary-count.test \
testsuite-summary-count-many.test \
testsuite-summary-reference-log.test \
-test-driver-end-metadata.test \
-test-driver-metadata-no-leading-space.test \
test-driver-custom-no-extra-driver.test \
test-driver-custom.test \
test-driver-custom-xfail-tests.test \
@@ -754,11 +754,17 @@ test-driver-custom-multitest-recheck2.test \
test-driver-custom-html.test \
test-driver-custom-no-html.test \
test-driver-create-log-dir.test \
-test-driver-global-log.test \
-test-driver-recheck.test \
test-driver-strip-vpath.test \
test-driver-fail.test \
test-log.test \
+test-metadata-global-log.test \
+test-metadata-recheck.test \
+test-metadata-results.test \
+test-missing.test \
+test-missing2.test \
+test-trs-basic.test \
+test-trs-recover.test \
+test-trs-recover2.test \
parse.test \
percent.test \
percent2.test \
diff --git a/tests/Makefile.in b/tests/Makefile.in
index 13e850c..43a57d5 100644
--- a/tests/Makefile.in
+++ b/tests/Makefile.in
@@ -124,10 +124,8 @@ am__rst_title = sed 's/.*/ &
/;h;s/./=/g;p;x;p;g;p;s/.*//'
# pass "-e" to $(SHELL), and POSIX 2008 even requires this. Work around it
# by disabling -e (using the XSI extension "set +e") if it's set.
am__sh_e_setup = case $$- in *e*) set +e;; esac
-# Default flags passed to all log compiler wrappers.
-am__test_driver_flags = \
- --test-name "$$f" \
- --log-file '$@' \
+# Default flags passed to test drivers.
+am__common_driver_flags = \
--color-tests "$$am__color_tests" \
--enable-hard-errors "$$am__enable_hard_errors" \
--expect-failure "$$am__expect_failure"
@@ -161,6 +159,14 @@ case " $(XFAIL_TESTS) " in \
am__expect_failure=no;; \
esac; \
$(AM_TESTS_ENVIRONMENT) $(TESTS_ENVIRONMENT)
+# The names of the tests, with any registered extension removed. Or
+# equivalently, the names of the test logs, with the `.log' extension
+# renoved. This honours runtime overriding of TESTS and TEST_LOGS.
+am__TEST_BASES = $(TEST_LOGS:.log=)
+# This can be used instead of $(MAKE) in recipes requiring a recursive call
+# to make, but which are not intended to be executed by "make -n". See the
+# GNU make manual for more details.
+am__stealth_MAKE = $(MAKE)
RECHECK_LOGS = $(TEST_LOGS)
AM_RECURSIVE_TARGETS = check check-html recheck recheck-html
TEST_SUITE_LOG = test-suite.log
@@ -969,9 +975,10 @@ parallel-tests7.test \
parallel-tests8.test \
parallel-tests9.test \
parallel-tests10.test \
+parallel-tests-make-n.test \
parallel-tests-fd-redirect.test \
parallel-tests-am_tests_environment.test \
-parallel-tests-unreadable-log.test \
+parallel-tests-unreadable.test \
parallel-tests-subdir.test \
parallel-tests-interrupt.test \
parallel-tests-reset-term.test \
@@ -984,12 +991,11 @@ parallel-test-driver-install.test \
parallel-tests-no-spurious-summary.test \
parallel-tests-exit-statuses.test \
parallel-tests-console-output.test \
+parallel-tests-once.test \
testsuite-summary-color.test \
testsuite-summary-count.test \
testsuite-summary-count-many.test \
testsuite-summary-reference-log.test \
-test-driver-end-metadata.test \
-test-driver-metadata-no-leading-space.test \
test-driver-custom-no-extra-driver.test \
test-driver-custom.test \
test-driver-custom-xfail-tests.test \
@@ -999,11 +1005,17 @@ test-driver-custom-multitest-recheck2.test \
test-driver-custom-html.test \
test-driver-custom-no-html.test \
test-driver-create-log-dir.test \
-test-driver-global-log.test \
-test-driver-recheck.test \
test-driver-strip-vpath.test \
test-driver-fail.test \
test-log.test \
+test-metadata-global-log.test \
+test-metadata-recheck.test \
+test-metadata-results.test \
+test-missing.test \
+test-missing2.test \
+test-trs-basic.test \
+test-trs-recover.test \
+test-trs-recover2.test \
parse.test \
percent.test \
percent2.test \
@@ -1396,7 +1408,7 @@ $(parallel_tests)
all: all-am
.SUFFIXES:
-.SUFFIXES: .html .log .test
+.SUFFIXES: .html .log .test .trs
$(srcdir)/Makefile.in: $(srcdir)/Makefile.am $(srcdir)/parallel-tests.am
$(srcdir)/instspc-tests.am $(am__configure_deps)
@for dep in $?; do \
case '$(am__configure_deps)' in \
@@ -1442,43 +1454,46 @@ CTAGS:
cscope cscopelist:
+# Recover from deleted `.trs' file; this should ensure that
+# "rm -f foo.log; make foo.trs" re-run `foo.test', and re-create
+# both `foo.log' and `foo.trs'.
+.log.trs:
+ rm -f $< $@ && $(am__stealth_MAKE) $(AM_MAKEFLAGS) $<
+
$(TEST_SUITE_LOG): $(TEST_LOGS)
@$(am__sh_e_setup); $(am__tty_colors); \
- tab=' '; \
- nlinit=`echo 'nl="'; echo '"'`; eval "$$nlinit"; unset nlinit; \
- IFS=" $$tab$$nl"; oIFS=$$IFS; \
- ws="[ $$tab]"; \
- list='$(TEST_LOGS)'; \
- list2=`for f in $$list; do test ! -r $$f || echo $$f; done`; \
- results1=`for f in $$list; do test -r $$f || echo ERROR; done`; \
- results2=''; \
- exec 5<&0; \
- for f in $$list2; do \
- exec 0<$$f; \
- IFS=''; \
- while read line; do \
- IFS=$$oIFS; \
- case $$line in \
- :end-metadata:*) \
- break;; \
- :test-result:*) \
- res=`expr "x$$line" : "x:test-result:$$ws*\\\\(.*\\\\)$$"`; \
- results2="$$results2$$nl$$res";; \
- esac; \
- IFS=''; \
+ bases='$(am__TEST_BASES)'; \
+ ws='[ ]'; \
+ redo_bases=`for b in $$bases; do \
+ test -f $$b.trs && test -r $$b.trs \
+ && test -f $$b.log && test -r $$b.log \
+ || echo $$b; \
+ done`; \
+ if test -n "$$redo_bases"; then \
+ redo_logs=`for i in $$redo_bases; do echo $$i.log; done`; \
+ redo_results=`for i in $$redo_bases; do echo $$i.trs; done`; \
+ rm -f $$redo_logs && rm -f $$redo_results \
+ && $(am__stealth_MAKE) $(AM_MAKEFLAGS) $$redo_logs; \
+ fi; \
+ st=0; \
+ for b in $$redo_bases; do \
+ for e in trs log; do \
+ if test ! -f $$b.$$e || test ! -r $$b.$$e; then \
+ echo "fatal: making $@: failed to create $$b.$$e" >&2; \
+ st=1; \
+ fi; \
done; \
- IFS=$$oIFS; \
- exec 0<&5; \
done; \
- exec 5<&-; \
- results=`echo "$$results1" && echo "$$results2"`; \
- all=`echo "$$results" | sed '/^$$/d' | wc -l | sed -e 's/^[
]*//'`; \
- fail=`echo "$$results" | grep -c '^FAIL'`; \
- pass=`echo "$$results" | grep -c '^PASS'`; \
- skip=`echo "$$results" | grep -c '^SKIP'`; \
- xfail=`echo "$$results" | grep -c '^XFAIL'`; \
- xpass=`echo "$$results" | grep -c '^XPASS'`; \
- error=`echo "$$results" | grep -c '^ERROR'`; \
+ test $$st -eq 0 || exit 1; \
+ results=`for b in $$bases; do echo $$b.trs; done`; \
+ test -n "$$results" || results=/dev/null; \
+ all=` grep "^$$ws*:test-result:" $$results | wc -l`; \
+ pass=` grep "^$$ws*:test-result:$$ws*PASS" $$results | wc -l`; \
+ fail=` grep "^$$ws*:test-result:$$ws*FAIL" $$results | wc -l`; \
+ skip=` grep "^$$ws*:test-result:$$ws*SKIP" $$results | wc -l`; \
+ xfail=`grep "^$$ws*:test-result:$$ws*XFAIL" $$results | wc -l`; \
+ xpass=`grep "^$$ws*:test-result:$$ws*XPASS" $$results | wc -l`; \
+ error=`grep "^$$ws*:test-result:$$ws*ERROR" $$results | wc -l`; \
if test `expr $$fail + $$xpass + $$error` -eq 0; then \
success=true; \
else \
@@ -1520,10 +1535,15 @@ $(TEST_SUITE_LOG): $(TEST_LOGS)
echo; \
echo ".. contents:: :depth: 2"; \
echo; \
- for f in $$list; do \
- sed -e '/^:end-metadata:/,$$d' $$f | \
- grep "^:copy-in-global-log:$$ws*no$$ws*$$" >/dev/null \
- || { echo && cat $$f; } \
+ for b in $$bases; do \
+ if grep "^$$ws*:copy-in-global-log:$$ws*no$$ws*$$" $$b.trs \
+ >/dev/null; then :; \
+ elif test ! -r $$b.log; then \
+ echo "ERROR: cannot read $$b.log" >&2; \
+ echo && echo "WARNING: could not read $$b.log!"; \
+ else \
+ echo && cat $$b.log; \
+ fi; \
done; \
} >$(TEST_SUITE_LOG).tmp; \
mv $(TEST_SUITE_LOG).tmp $(TEST_SUITE_LOG); \
@@ -1550,6 +1570,7 @@ $(TEST_SUITE_LOG): $(TEST_LOGS)
# Run all the tests.
check-TESTS:
@list='$(RECHECK_LOGS)'; test -z "$$list" || rm -f $$list
+ @list='$(RECHECK_LOGS:.log=.trs)'; test -z "$$list" || rm -f $$list
@test -z "$(TEST_SUITE_LOG)" || rm -f $(TEST_SUITE_LOG)
@list='$(TEST_LOGS)'; \
list=`for f in $$list; do \
@@ -1584,19 +1605,19 @@ check-html:
recheck recheck-html:
@ws='[ ]'; \
target=`echo $@ | sed 's,^re,,'`; \
- list='$(TEST_LOGS)'; \
- list=`for f in $$list; do \
- test -f $$f || continue; \
- sed -e '/^:end-metadata:/,$$d' $$f | \
- grep "^:recheck:$$ws*no$$ws*$$" >/dev/null \
- || echo $$f; \
+ bases='$(am__TEST_BASES)'; \
+ list=`for b in $$bases; do \
+ test ! -f $$b.trs && test ! -f $$b.log && continue; \
+ grep "^$$ws*:recheck:$$ws*no$$ws*$$" $$b.trs \
+ >/dev/null 2>&1 || echo $$b.log; \
done | tr '\012\015' ' '`; \
list=`echo "$$list" | sed 's/ *$$//'`; \
$(MAKE) $(AM_MAKEFLAGS) $$target AM_MAKEFLAGS='$(AM_MAKEFLAGS)
TEST_LOGS="'"$$list"'"'
.test.log:
- @p='$<'; $(am__check_pre) \
- $(TEST_LOG_DRIVER) $(am__test_driver_flags) $(AM_TEST_LOG_DRIVER_FLAGS)
$(TEST_LOG_DRIVER_FLAGS) -- \
- $(TEST_LOG_COMPILE) "$$tst" $(AM_TESTS_FD_REDIRECT)
+ @p='$<'; $(am__check_pre) $(TEST_LOG_DRIVER) --test-name "$$f" \
+ --log-file '$*.log' --result-file '$*.trs' \
+ $(am__common_driver_flags) $(AM_TEST_LOG_DRIVER_FLAGS)
$(TEST_LOG_DRIVER_FLAGS) -- $(TEST_LOG_COMPILE) "$$tst" \
+ $(AM_TESTS_FD_REDIRECT)
distdir: $(DISTFILES)
@srcdirstrip=`echo "$(srcdir)" | sed 's/[].[^$$\\*]/\\\\&/g'`; \
@@ -1657,6 +1678,7 @@ mostlyclean-generic:
-test -z "$(TEST_LOGS_TMP)" || rm -f $(TEST_LOGS_TMP)
-test -z "$(TEST_SUITE_HTML)" || rm -f $(TEST_SUITE_HTML)
-test -z "$(TEST_SUITE_LOG)" || rm -f $(TEST_SUITE_LOG)
+ -test -z "$(am__TEST_BASES:=.trs)" || rm -f $(am__TEST_BASES:=.trs)
clean-generic:
diff --git a/tests/parallel-tests-make-n.test b/tests/parallel-tests-make-n.test
new file mode 100755
index 0000000..451f782
--- /dev/null
+++ b/tests/parallel-tests-make-n.test
@@ -0,0 +1,107 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# Check `make -n' for testsuite-related targets, when `parallel-tests'
+# is in use.
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cat >> configure.in << 'END'
+AC_OUTPUT
+END
+
+cat > Makefile.am <<'END'
+TESTS = foo.test bar.test
+TEST_LOG_COMPILER = $(SHELL)
+END
+
+$ACLOCAL
+$AUTOMAKE -a
+$AUTOCONF
+
+./configure
+
+make_n_ ()
+{
+ st=0
+ $MAKE -n "$@" >output 2>&1 || { cat output; ls -l; Exit 1; }
+ cat output
+ # Look out for possible errors from common tools used by recipes.
+ $EGREP -i ' (exist|permission|denied|no .*(such|file))' output && Exit 1
+ $EGREP '(mv|cp|rm|cat|grep|sed|awk): ' output && Exit 1
+ :
+}
+
+: > output
+
+files='foo.log bar.log foo.trs bar.trs'
+
+for target in check recheck test-suite.log; do
+ test ! -f foo.log
+ test ! -f foo.trs
+ test ! -f bar.log
+ test ! -f bar.trs
+ test ! -f test-suite.log
+done
+
+echo 'exit 0' > foo.test
+echo 'exit 1' > bar.test
+
+$MAKE check && Exit 1
+
+chmod a-w .
+
+make_n_ clean
+test -f foo.log
+test -f foo.trs
+test -f foo.log
+test -f bar.trs
+
+echo 'echo this is bad; exit 1' > foo.test
+echo 'exit 0' > bar.test
+
+for target in check recheck test-suite.log; do
+ make_n_ $target
+ grep '^:test-result: *FAIL' bar.trs
+ grep 'this is bad' foo.log test-suite.log && Exit 1
+done
+
+chmod a-rw $files
+if test -r foo.log; then
+ : You can still read unreadable files! Skip these checks.
+else
+ for target in check recheck test-suite.log; do
+ make_n_ $target
+ for f in $files; do
+ test -f $f && test ! -r $f || Exit 1
+ done
+ done
+fi
+chmod u+r $files
+
+chmod u+w .
+rm -f foo.log bar.trs
+chmod a-w .
+for target in check recheck test-suite.log $files; do
+ make_n_ $target
+ test ! -f foo.log
+ test -f foo.trs
+ test ! -f bar.trs
+ test -f bar.log
+done
+
+:
diff --git a/tests/parallel-tests-unreadable-log.test
b/tests/parallel-tests-once.test
similarity index 59%
copy from tests/parallel-tests-unreadable-log.test
copy to tests/parallel-tests-once.test
index e8e101b..d676a12 100755
--- a/tests/parallel-tests-unreadable-log.test
+++ b/tests/parallel-tests-once.test
@@ -14,7 +14,9 @@
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
-# Check that the testsuite driver copes well with unreadable test logs.
+# Tests shouldn't be run multiple times by a simple "make check" in a
+# clean directory. An early implementation of the `.trs' intermediate
+# files incurred a similar problem.
parallel_tests=yes
. ./defs || Exit 1
@@ -24,37 +26,23 @@ AC_OUTPUT
END
cat > Makefile.am << 'END'
-TESTS = foo.test bar.test
-XFAIL_TESTS = bar.test
+TESTS = foo.test
END
-cat > foo.test << 'END'
+cat > foo.test <<'END'
#! /bin/sh
-exit 0
+test -f foo.run && Exit 1
+: > foo.run
END
-cat > bar.test << 'END'
-#! /bin/sh
-exit 1
-END
-chmod a+x foo.test bar.test
+chmod a+x foo.test
$ACLOCAL
$AUTOCONF
$AUTOMAKE -a
-# The testsuite driver will use this variable, so ensure it sanitizes
-# it and do not allow in spurious values from the environment.
-line=PASS; export line
-
./configure
-$MAKE foo.log
-$MAKE bar.log
-chmod a-r foo.log bar.log
-test ! -r foo.log || Exit 77
-$MAKE test-suite.log >stdout && { cat stdout; Exit 1; }
-cat stdout
-grep '^# ERROR: *2$' stdout
-grep '^# ERROR: *2$' test-suite.log
+$MAKE check
+test -f foo.run # Sanity check.
:
diff --git a/tests/parallel-tests-unreadable-log.test
b/tests/parallel-tests-unreadable.test
similarity index 58%
copy from tests/parallel-tests-unreadable-log.test
copy to tests/parallel-tests-unreadable.test
index e8e101b..47edc9a 100755
--- a/tests/parallel-tests-unreadable-log.test
+++ b/tests/parallel-tests-unreadable.test
@@ -14,47 +14,62 @@
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
-# Check that the testsuite driver copes well with unreadable test logs.
+# Check that the testsuite driver copes well with unreadable `.log'
+# and `.trs' files.
parallel_tests=yes
. ./defs || Exit 1
+: > t
+chmod a-r t && test ! -r t || skip_ "you can still read unreadable files"
+rm -f t
+
cat >> configure.in << 'END'
AC_OUTPUT
END
cat > Makefile.am << 'END'
TESTS = foo.test bar.test
-XFAIL_TESTS = bar.test
END
cat > foo.test << 'END'
#! /bin/sh
+echo foofoofoo
exit 0
END
+
cat > bar.test << 'END'
#! /bin/sh
-exit 1
+echo barbarbar
+exit 77
END
+
chmod a+x foo.test bar.test
$ACLOCAL
$AUTOCONF
$AUTOMAKE -a
-# The testsuite driver will use this variable, so ensure it sanitizes
-# it and do not allow in spurious values from the environment.
-line=PASS; export line
-
./configure
-$MAKE foo.log
-$MAKE bar.log
-chmod a-r foo.log bar.log
-test ! -r foo.log || Exit 77
-$MAKE test-suite.log >stdout && { cat stdout; Exit 1; }
-cat stdout
-grep '^# ERROR: *2$' stdout
-grep '^# ERROR: *2$' test-suite.log
+for files in \
+ 'foo.log bar.log' \
+ 'foo.trs bar.trs' \
+ 'foo.trs bar.log' \
+ 'foo.log bar.trs' \
+; do
+ $MAKE check
+ rm -f test-suite.log
+ chmod a-r $files
+ $MAKE test-suite.log || { ls -l; Exit 1; }
+ ls -l
+ grep '^foofoofoo$' foo.log
+ grep '^:test-result: PASS' foo.trs
+ grep '^barbarbar$' bar.log
+ grep '^:test-result: SKIP' bar.trs
+ grep '^SKIP: bar.test' test-suite.log
+ grep '^barbarbar$' test-suite.log
+ $EGREP 'foo\.test|foofoofoo' test-suite.log && Exit 1
+done
:
diff --git a/tests/parallel-tests9.test b/tests/parallel-tests9.test
index 68fd1d4..f86562a 100755
--- a/tests/parallel-tests9.test
+++ b/tests/parallel-tests9.test
@@ -67,15 +67,6 @@ grep 'foo\.test' stdout && Exit 1
grep '^ERROR: bar\.test$' stdout
grep '^FAIL: baz\.test$' stdout
-# If we cannot read the log file, then redo it as well.
-chmod a-r foo.log
-if test ! -r foo.log; then
- $MAKE recheck >stdout && { cat stdout; Exit 1; }
- cat stdout
- count_test_results total=3 pass=1 fail=1 skip=0 xfail=0 xpass=0 error=1
- grep '^PASS: foo\.test$' stdout || Exit 1
-fi
-
# Ensure that recheck builds check_SCRIPTS, and that
# recheck reruns nothing if check has not been run.
$MAKE clean
diff --git a/tests/test-driver-create-log-dir.test
b/tests/test-driver-create-log-dir.test
index 5713b92..a7f83ba 100755
--- a/tests/test-driver-create-log-dir.test
+++ b/tests/test-driver-create-log-dir.test
@@ -40,14 +40,19 @@ check-local: $(TEST_SUITE_LOG)
test -f sub/foo.log
test -f sub/subsub/bar.log
test -f sub1/baz.log
+ test -f sub/foo.trs
+ test -f sub/subsub/bar.trs
+ test -f sub1/baz.trs
END
cat > checkdir-driver <<'END'
#! /bin/sh
set -e
+set -u
while test $# -gt 0; do
case $1 in
--log-file) log_file=$2; shift;;
+ --result-file) trs_file=$2; shift;;
--test-name|--expect-failure|--color-tests|--enable-hard-errors) shift;;
--) shift; break;;
*) echo "$0: invalid option/argument: '$1'" >&2; exit 2;;
@@ -55,11 +60,16 @@ while test $# -gt 0; do
shift
done
echo "log: $log_file" # For debugging.
+echo "trs: $trs_file" # Ditto.
case $log_file in */*);; *) exit 1;; esac
-dir=`expr "$log_file" : '\(.*\)/[^/]*'`
-echo "dir: $dir" # For debugging.
-test -d "$dir" || exit 1
-echo dummy > "$log_file"
+dir_log=`expr "$log_file" : '\(.*\)/[^/]*'`
+dir_trs=`expr "$trs_file" : '\(.*\)/[^/]*'`
+echo "dir_log: $dir_log" # For debugging.
+echo "dir_trs: $dir_trs" # Likewise.
+test x"$dir_trs" = x"$dir_log" || exit 1
+test -d "$dir_log" || exit 1
+echo dummy1 > "$log_file"
+echo dummy2 > "$trs_file"
END
chmod a+x checkdir-driver
diff --git a/tests/test-driver-custom-multitest.test
b/tests/test-driver-custom-multitest.test
index df0568c..4fb023c 100755
--- a/tests/test-driver-custom-multitest.test
+++ b/tests/test-driver-custom-multitest.test
@@ -127,6 +127,7 @@ for vpath in : false; do
# `trivial-test-driver' script is changed.
$FGREP INVALID.NAME stdout test-suite.log && Exit 1
test -f BAD.LOG && Exit 1
+ test -f BAD.TRS && Exit 1
# These log files must all have been created by the testsuite.
cat pass.log
cat fail.log
diff --git a/tests/test-driver-custom-no-html.test
b/tests/test-driver-custom-no-html.test
index 683659b..1a2b726 100755
--- a/tests/test-driver-custom-no-html.test
+++ b/tests/test-driver-custom-no-html.test
@@ -35,14 +35,14 @@ END
cat > no-rst <<'END'
#! /bin/sh
+echo ':test-result: SKIP' > foo.trs
+echo ':copy-in-global-log: yes' >> foo.trs
# The genereted log file is deliberately syntactically invalid
# reStructuredText.
cat > foo.log <<'EoL'
SKIP: FooBar
=============
-:test-result: SKIP
-
--------------
dummy title
EoL
diff --git a/tests/test-driver-custom-xfail-tests.test
b/tests/test-driver-custom-xfail-tests.test
index 4f8bb83..aabf2f1 100755
--- a/tests/test-driver-custom-xfail-tests.test
+++ b/tests/test-driver-custom-xfail-tests.test
@@ -88,15 +88,17 @@ done
cat > td <<'END'
#! /bin/sh
-set -e
+set -e; set -u
test_name=INVALID
log_file=/dev/null
+trs_file=/dev/null
expect_failure=no
while test $# -gt 0; do
case $1 in
--test-name) test_name=$2; shift;;
--expect-failure) expect_failure=$2; shift;;
--log-file) log_file=$2; shift;;
+ --result-file) trs_file=$2; shift;;
# Ignored.
--color-tests) shift;;
--enable-hard-errors) shift;;
@@ -109,23 +111,22 @@ while test $# -gt 0; do
done
st=0
"$@" || st=$?
-rm -f "$log_file"
case $st,$expect_failure in
0,no)
echo "PASS: $test_name" | tee "$log_file"
- echo ":test-result: PASS" >> "$log_file"
+ echo ":test-result: PASS" > "$trs_file"
;;
1,no)
echo "FAIL: $test_name" | tee "$log_file"
- echo ":test-result: FAIL" >> "$log_file"
+ echo ":test-result: FAIL" > "$trs_file"
;;
0,yes)
echo "XPASS: $test_name" | tee "$log_file"
- echo ":test-result: XPASS" >> "$log_file"
+ echo ":test-result: XPASS" > "$trs_file"
;;
1,yes)
echo "XFAIL: $test_name" | tee "$log_file"
- echo ":test-result: XFAIL" >> "$log_file"
+ echo ":test-result: XFAIL" > "$trs_file"
;;
*)
echo "INTERNAL ERROR" >&2
@@ -141,21 +142,24 @@ $AUTOMAKE
./configure
-VERBOSE=yes $MAKE check
-test -f test-suite.log
-test -f sub1/test-suite.log
-test -f sub2/test-suite.log
+$MAKE check >stdout || { cat stdout; Exit 1; }
+cat stdout
+test `grep -c '^PASS:' stdout` -eq 3
+test `grep -c '^XFAIL:' stdout` -eq 13
for dir in sub1 sub2; do
cd $dir
cp pass.test x1.test
cp x2.test pass.test
- $MAKE check && { cat test-suite.log; Exit 1; }
- cat test-suite.log
- grep '^FAIL: pass\.test$' test-suite.log
- grep '^XPASS: x1\.test$' test-suite.log
- test `grep -c '^FAIL' test-suite.log` -eq 1
- test `grep -c '^XPASS' test-suite.log` -eq 1
+ $MAKE check >stdout && { cat stdout; Exit 1; }
+ cat stdout
+ test "`cat pass.trs`" = ":test-result: FAIL"
+ test "`cat x1.trs`" = ":test-result: XPASS"
+ test "`cat x2.trs`" = ":test-result: XFAIL"
+ grep '^FAIL: pass\.test$' stdout
+ grep '^XPASS: x1\.test$' stdout
+ grep '^XFAIL: x2\.test$' stdout
+ count_test_results total=7 pass=0 xpass=1 fail=1 xfail=5 skip=0 error=0
cd ..
done
diff --git a/tests/test-driver-custom.test b/tests/test-driver-custom.test
index b6a33d2..dec6aee 100755
--- a/tests/test-driver-custom.test
+++ b/tests/test-driver-custom.test
@@ -59,11 +59,13 @@ log_wflags='@log_wflags@'
test_name=INVALID
log_file=BAD.log
+trs_file=BAD.trs
extra_opts=
while test $# -gt 0; do
case $1 in
--test-name) test_name=$2; shift;;
--log-file) log_file=$2; shift;;
+ --result-file) trs_file=$2; shift;;
# Ignored.
--expect-failure) shift;;
--color-tests) shift;;
@@ -79,6 +81,7 @@ while test $# -gt 0; do
done
echo "$me" "$test_name" $extra_opts > "$log_file"
+: > "$trs_file"
exec "$@"
exit 127
@@ -123,6 +126,7 @@ VERBOSE=yes $MAKE check
ls -l . sub
test ! -r BAD.log
+test ! -r BAD.trs
echo 'chk-wrapper 1.chk --am-chk --chk' > 1.exp
echo 'test-wrapper 2.test -am-test -test' > 2.exp
diff --git a/tests/test-driver-end-metadata.test
b/tests/test-driver-end-metadata.test
deleted file mode 100755
index 42e7e57..0000000
--- a/tests/test-driver-end-metadata.test
+++ /dev/null
@@ -1,120 +0,0 @@
-#! /bin/sh
-# Copyright (C) 2011 Free Software Foundation, Inc.
-#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation; either version 2, or (at your option)
-# any later version.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with this program. If not, see <http://www.gnu.org/licenses/>.
-
-# Custom test drivers and parallel-tests harness: check the documented
-# semantics for inhibiting the scanning of the remaining portion of a
-# `.log' file (for what concerns metadata). Currently, this is done
-# with the use of the reStructuredText field `:end-metadata:'.
-
-parallel_tests=yes
-. ./defs || Exit 1
-
-cat >> configure.in << 'END'
-AC_OUTPUT
-END
-
-cat > Makefile.am << 'END'
-TEST_LOG_DRIVER = ./passthrough-driver
-TESTS = foo.test bar.test baz.test
-END
-
-cat > passthrough-driver <<'END'
-#!/bin/sh
-set -e; set -u;
-while test $# -gt 0; do
- case $1 in
- --log-file) log_file=$2; shift;;
- --test-name) test_name=$2; shift;;
- --expect-failure|--color-tests|--enable-hard-errors) shift;;
- --) shift; break;;
- *) echo "$0: invalid option/argument: '$1'" >&2; exit 2;;
- esac
- shift
-done
-echo "$test_name: RUN"
-cp $1 $log_file
-END
-chmod a+x passthrough-driver
-
-cat > foo.test <<END
-:end-metadata:
-:test-result: FAIL
-:copy-in-global-log: no
-:recheck: no
-foo foo foo
-END
-
-cat > bar.test <<END
-:test-result: PASS
-:recheck: no
-bar bar bar
-:copy-in-global-log: no
-:end-metadata: $tab
-:test-result: ERROR
-:end-metadata:
-END
-
-# `:end-metadata:' directives in other files shouldn't influence this one.
-cat > baz.test <<END
-:test-result: PASS
-:recheck: no
-:copy-in-global-log: no
-baz baz baz
-END
-
-$ACLOCAL
-$AUTOCONF
-$AUTOMAKE
-
-./configure
-
-st=0
-$MAKE check >stdout || st=1
-# For debugging.
-cat stdout
-cat foo.log
-cat bar.log
-cat test-suite.log
-
-# Check that only the `:test-result:' in baz.test and the first one in
-# foo.test have been considered.
-test $st -eq 0
-grep '^# TOTAL: *2$' stdout
-grep '^# PASS: *2$' stdout
-for result in SKIP XPASS FAIL XFAIL ERROR; do
- grep "^# $result: *0$" stdout
-done
-
-# Check that a `:end-metadata:' directive does not prevent following text
-# to be copied into the log file.
-$FGREP 'foo foo foo' foo.log
-$FGREP 'bar bar bar' bar.log
-$FGREP 'baz baz baz' baz.log # More of a sanity check really.
-
-# Check that only the `:copy-in-global-log:' in bar.test has been
-# considered.
-$FGREP 'foo foo foo' test-suite.log
-$FGREP 'bar bar bar' test-suite.log && Exit 1
-$FGREP 'baz baz baz' test-suite.log && Exit 1
-
-# Check that only the `:recheck:' in bar.test has been
-# considered.
-$MAKE recheck >stdout || { cat stdout; Exit 1; }
-cat stdout
-grep '^foo\.test: RUN$' stdout
-grep 'ba[rz]\.test.*RUN' stdout && Exit 1
-
-:
diff --git a/tests/test-driver-metadata-no-leading-space.test
b/tests/test-driver-metadata-no-leading-space.test
deleted file mode 100755
index 4c3f026..0000000
--- a/tests/test-driver-metadata-no-leading-space.test
+++ /dev/null
@@ -1,94 +0,0 @@
-#! /bin/sh
-# Copyright (C) 2011 Free Software Foundation, Inc.
-#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation; either version 2, or (at your option)
-# any later version.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with this program. If not, see <http://www.gnu.org/licenses/>.
-
-# Custom test drivers and parallel-tests harness: check that, in the
-# log files, the reStructuredText fields for metadata declaration are
-# recognized only when placed at the beginning of a line.
-
-parallel_tests=yes
-. ./defs || Exit 1
-
-cat >> configure.in << 'END'
-AC_OUTPUT
-END
-
-cat > Makefile.am << 'END'
-TEST_LOG_DRIVER = ./passthrough-driver
-TESTS = foo.test
-END
-
-cat > passthrough-driver <<'END'
-#!/bin/sh
-set -e; set -u;
-while test $# -gt 0; do
- case $1 in
- --log-file) log_file=$2; shift;;
- --test-name) test_name=$2; shift;;
- --expect-failure|--color-tests|--enable-hard-errors) shift;;
- --) shift; break;;
- *) echo "$0: invalid option/argument: '$1'" >&2; exit 2;;
- esac
- shift
-done
-echo "$test_name: RUN"
-cp $1 $log_file
-END
-chmod a+x passthrough-driver
-
-cat > foo.test <<END
- $tab :test-result: FAIL
- $tab :end-metadata:
-:test-result: PASS
- $tab :copy-in-global-log: no
- $tab :recheck: no
-foo foo foo
-END
-
-$ACLOCAL
-$AUTOCONF
-$AUTOMAKE
-
-./configure
-
-st=0
-$MAKE check >stdout || st=1
-# For debugging.
-cat stdout
-cat foo.log
-cat test-suite.log
-
-# Check that only the second `:test-result:' has been processed.
-test $st -eq 0
-grep '^# TOTAL: *1$' stdout
-grep '^# PASS: *1$' stdout
-for result in SKIP XPASS FAIL XFAIL ERROR; do
- grep "^# $result: *0$" stdout
-done
-
-# Check that `:copy-in-global-log:' hasn't been processed.
-grep 'foo foo foo' test-suite.log
-
-# Check that `:recheck:' hasn't been processed.
-: > older
-$sleep
-$MAKE recheck >stdout || { cat stdout; Exit 1; }
-cat stdout
-grep '^foo\.test: RUN$' stdout
-grep 'foo foo foo' test-suite.log
-is_newest test-suite.log older
-is_newest foo.log older
-
-:
diff --git a/tests/test-driver-strip-vpath.test
b/tests/test-driver-strip-vpath.test
index f1b316c..c2b3113 100755
--- a/tests/test-driver-strip-vpath.test
+++ b/tests/test-driver-strip-vpath.test
@@ -51,6 +51,7 @@ set -e
while test $# -gt 0; do
case $1 in
--log-file) log_file=$2; shift;;
+ --result-file) trs_file=$2; shift;;
--test-name) test_name=$2; shift;;
--expect-failure|--color-tests|--enable-hard-errors) shift;;
--) shift; break;;
@@ -64,6 +65,7 @@ case $test_name in
*) exit 1;;
esac
echo dummy > "$log_file"
+echo dummy > "$trs_file"
END
chmod a+x checkstrip-driver
diff --git a/tests/parallel-tests-unreadable-log.test
b/tests/test-driver-trs-suffix-registered.test
similarity index 51%
copy from tests/parallel-tests-unreadable-log.test
copy to tests/test-driver-trs-suffix-registered.test
index e8e101b..7f6522a 100755
--- a/tests/parallel-tests-unreadable-log.test
+++ b/tests/test-driver-trs-suffix-registered.test
@@ -14,47 +14,45 @@
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
-# Check that the testsuite driver copes well with unreadable test logs.
+# parallel-tests support: the following are registered with `.SUFFIXES':
+# - .log
+# - .trs (used by files that store test results and metadata)
+# - .test if $(TEST_EXTENSIONS) is not defined
+# - stuff in $(TEST_EXTENSIONS) otherwise
parallel_tests=yes
. ./defs || Exit 1
-cat >> configure.in << 'END'
-AC_OUTPUT
-END
+: > Makefile.am
-cat > Makefile.am << 'END'
-TESTS = foo.test bar.test
-XFAIL_TESTS = bar.test
+cat > 1.am << 'END'
+TESTS =
END
-cat > foo.test << 'END'
-#! /bin/sh
-exit 0
+cat > 2.am << 'END'
+TEST_EXTENSIONS = .SH .abcdef
+TESTS =
END
-cat > bar.test << 'END'
-#! /bin/sh
-exit 1
-END
-chmod a+x foo.test bar.test
+
+: > test-driver
$ACLOCAL
-$AUTOCONF
-$AUTOMAKE -a
-
-# The testsuite driver will use this variable, so ensure it sanitizes
-# it and do not allow in spurious values from the environment.
-line=PASS; export line
-
-./configure
-
-$MAKE foo.log
-$MAKE bar.log
-chmod a-r foo.log bar.log
-test ! -r foo.log || Exit 77
-$MAKE test-suite.log >stdout && { cat stdout; Exit 1; }
-cat stdout
-grep '^# ERROR: *2$' stdout
-grep '^# ERROR: *2$' test-suite.log
+
+$AUTOMAKE 1
+$AUTOMAKE 2
+
+sed -e 's/$/ /' 1.in > mk.1
+sed -e 's/$/ /' 2.in > mk.2
+
+grep '^\.SUFFIXES:' mk.1
+grep '^\.SUFFIXES:' mk.2
+
+for suf in test log trs; do
+ grep "^\\.SUFFIXES:.* \\.$suf " mk.1
+done
+
+for suf in SH abcdef log trs; do
+ grep "^\\.SUFFIXES:.* \\.$suf " mk.2
+done
:
diff --git a/tests/test-driver-global-log.test
b/tests/test-metadata-global-log.test
similarity index 62%
rename from tests/test-driver-global-log.test
rename to tests/test-metadata-global-log.test
index 7596e94..c949bce 100755
--- a/tests/test-driver-global-log.test
+++ b/tests/test-metadata-global-log.test
@@ -17,7 +17,8 @@
# Custom test drivers and parallel-tests harness: check the documented
# semantics for deciding when the content of a test log file should be
# copied in the global test-suite.log file. Currently, this is done
-# with the use of the reStructuredText field `:copy-in-global-log:'.
+# with the use of the reStructuredText field `:copy-in-global-log:' in
+# the associated `.trs' files.
parallel_tests=yes
. ./defs || Exit 1
@@ -26,86 +27,104 @@ cat >> configure.in << 'END'
AC_OUTPUT
END
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = ./passthrough-driver
+TEST_LOG_COMPILER = $(SHELL) -e
+END
+
+cat > passthrough-driver <<'END'
+#!/bin/sh
+set -e; set -u;
+while test $# -gt 0; do
+ case $1 in
+ --log-file) log_file=$2; shift;;
+ --result-file) trs_file=$2; shift;;
+ --test-name) test_name=$2; shift;;
+ --expect-failure|--color-tests|--enable-hard-errors) shift;;
+ --) shift; break;;
+ *) echo "$0: invalid option/argument: '$1'" >&2; exit 2;;
+ esac
+ shift
+done
+echo "$test_name: RUN"
+"$@" >$log_file 2>&1 5>$trs_file
+END
+chmod a+x passthrough-driver
+
# The `:test-result:' and `:recheck:' fields and the first line of the
# log file should be be irrelevant for the decision of whether a test
# output is to be copied in the `test-suite.log'.
cat > no-1.test <<END
-FAIL: no-1.test
-:test-result: SKIP
-:copy-in-global-log: no
-:test-result: FAIL
-:test-result: XPASS
-not seen 1
+echo :test-result: SKIP >&5
+echo :copy-in-global-log: no >&5
+echo :test-result: FAIL >&5
+echo :test-result: XPASS >&5
+echo not seen 1
+END
+
+# In the last line, with leading and trailing whitespace in the value.
+cat > no-2.test <<END
+echo ":test-result: FAIL" >&5
+echo "not seen 2"
+echo ":recheck: yes" >&5
+echo ":copy-in-global-log:$tab $tab no $tab" >&5
END
for RES in XPASS FAIL XFAIL SKIP ERROR UNKNOWN; do
unindent > $RES.test <<END
- $RES: $RES.test
- :test-result: $RES
- :copy-in-global-log: no
- not seen $RES
+ echo :test-result: $RES >&5
+ echo :copy-in-global-log: no >&5
+ echo not seen $RES
END
done
-# In the last line, with leading and trailing whitespace.
-cat > no-2.test <<END
-:test-result: FAIL
-not seen 2
-:recheck: yes
-:copy-in-global-log:$tab $tab no $tab
-END
-
# In the first line, with no whitespace.
cat > no-3.test <<END
-:copy-in-global-log:no
-not seen 3
-:test-result: FAIL
+echo :copy-in-global-log:no >&5
+echo ":test-result: FAIL" >&5
+echo "not seen 3"
+END
+
+# Leading whitespace before the field.
+cat > no-4.test <<END
+echo ":test-result: FAIL" >&5
+echo " $tab $tab$tab :copy-in-global-log: no" >&5
+echo "not seen 4"
END
cat > yes-1.test <<END
-PASS: yes-1.test
-:test-result: PASS
-:copy-in-global-log: yes
-seen yes 1
+echo :test-result: PASS >&5
+echo :copy-in-global-log: yes >&5
+echo seen yes 1
END
# A lacking `:copy-in-global-log:' implies that the content of
# the log file should be copied.
cat > yes-2.test <<END
-PASS: y.test
-:test-result: PASS
-seen yes 2
+echo :test-result: PASS >&5
+echo seen yes 2
END
-# Corner cases.
-echo ':copy-in-global-log:' > corn-1.test
-echo " $tab $tab$tab" > corn-2.test
-cat > corn-3.test <<'END'
-seen corn 31
-:copy-in-global-log:address@hidden
-seen corn 32
+# Three corner cases.
+
+cat > corn-1.test <<END
+echo seen corn 1
+echo ':copy-in-global-log:' >&5
END
-echo TEST_LOG_DRIVER = ./dummy-driver > Makefile.am
-echo TESTS = *.test >> Makefile.am
+cat > corn-2.test <<END
+echo seen corn 2
+echo '$tab $tab$tab' >&5
+END
-cat > dummy-driver <<'END'
-#!/bin/sh
-set -e
-while test $# -gt 0; do
- case $1 in
- --log-file) log_file=$2; shift;;
- --test-name) test_name=$2; shift;;
- --expect-failure|--color-tests|--enable-hard-errors) shift;;
- --) shift; break;;
- *) echo "$0: invalid option/argument: '$1'" >&2; exit 2;;
- esac
- shift
-done
-cp $1 $log_file
+cat > corn-3.test <<'END'
+echo seen corn 31
+echo ':copy-in-global-log:address@hidden' >&5
+echo seen corn 32
END
-chmod a+x dummy-driver
+
+echo TESTS = *.test >> Makefile.am
$ACLOCAL
$AUTOCONF
@@ -117,15 +136,12 @@ $AUTOMAKE
# should be checked in other tests.
$MAKE check || :
cat test-suite.log
-
grep '^seen yes 1$' test-suite.log
grep '^seen yes 2$' test-suite.log
+grep '^seen corn 1$' test-suite.log
+grep '^seen corn 2$' test-suite.log
grep '^seen corn 31$' test-suite.log
grep '^seen corn 32$' test-suite.log
-grep '^:copy-in-global-log:$' test-suite.log
-grep "^ $tab $tab$tab$" test-suite.log
$FGREP 'not seen' test-suite.log && Exit 1
-$EGREP '^(XPASS|FAIL|SKIP|ERROR|UNKNOWN)' test-suite.log && Exit 1
-test `grep -c ':test-result:' test-suite.log` -eq 2
:
diff --git a/tests/test-driver-recheck.test b/tests/test-metadata-recheck.test
similarity index 93%
rename from tests/test-driver-recheck.test
rename to tests/test-metadata-recheck.test
index a8654c1..462a32e 100755
--- a/tests/test-driver-recheck.test
+++ b/tests/test-metadata-recheck.test
@@ -15,9 +15,7 @@
# along with this program. If not, see <http://www.gnu.org/licenses/>.
# Test the "make recheck" semantics for custom test drivers, as documented
-# in the Automake manual. Currently, that is based on the reStructuredText
-# field `:recheck:'.
-
+# in the Automake manual.
parallel_tests=yes
. ./defs || Exit 1
@@ -44,12 +42,12 @@ echo ":recheck: who cares" > y-5
echo ":recheck: $tab y" > y-6
echo ":recheck: yeah!$tab$tab " > y-7
cat > y-10 <<END
-foo
+:foo:
:recheck: ???
END
cat > y-11 <<END
:recheck: YES
-foo
+:foo:
END
cat > y-12 <<END
foo
@@ -59,6 +57,8 @@ bar
zardoz
END
+echo " $tab $tab$tab :recheck: yes" > y-8
+
# The :test-result: fields and the fist line of the log should be
# irrelevant for the decision of whether "make recheck" should or
# should not re-run a test.
@@ -83,12 +83,12 @@ echo ":recheck: no " > n-2
echo ":recheck: $tab no" > n-3
echo ":recheck: no $tab$tab " > n-4
cat > n-5 <<END
-foo
+:foo:
:recheck:no
END
cat > n-6 <<END
:recheck: no
-foo
+:foo:
END
cat > n-7 <<END
foo
@@ -98,6 +98,8 @@ bar
zardoz
END
+echo " $tab $tab$tab :recheck: no" > n-8
+
# The :test-result: fields should be irrelevant for the decision of
# whether "make recheck" should or should not re-run a test.
cat > n-100 <<END
@@ -117,10 +119,11 @@ cat Makefile.am # For debugging.
cat > dummy-driver <<'END'
#!/bin/sh
-set -e
+set -e; set -u
while test $# -gt 0; do
case $1 in
--log-file) log_file=$2; shift;;
+ --result-file) trs_file=$2; shift;;
--test-name) test_name=$2; shift;;
--expect-failure|--color-tests|--enable-hard-errors) shift;;
--) shift; break;;
@@ -129,7 +132,8 @@ while test $# -gt 0; do
shift
done
: > $test_name.run
-cp $1 $log_file
+: > $log_file
+cp $1 $trs_file
END
chmod a+x dummy-driver
diff --git a/tests/test-metadata-results.test b/tests/test-metadata-results.test
new file mode 100755
index 0000000..0cd97b7
--- /dev/null
+++ b/tests/test-metadata-results.test
@@ -0,0 +1,169 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# Parallel testsuite harness: check APIs for the registering of test
+# results in `*.trs' files, as documented in the automake manual.
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cat >> configure.in << 'END'
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = ./dummy-driver
+TESTS = foo.test bar.test
+END
+
+cat > dummy-driver <<'END'
+#! /bin/sh
+set -e; set -u
+while test $# -gt 0; do
+ case $1 in
+ --log-file) log_file=$2; shift;;
+ --result-file) trs_file=$2; shift;;
+ --test-name) test_name=$2; shift;;
+ --expect-failure|--color-tests|--enable-hard-errors) shift;;
+ --) shift; break;;
+ *) echo "$0: invalid option/argument: '$1'" >&2; exit 2;;
+ esac
+ shift
+done
+: > $log_file
+cp $1 $trs_file
+END
+chmod a+x dummy-driver
+
+mk_check ()
+{
+ st=0
+ $MAKE check >stdout || st=$?
+ cat stdout
+ # Our dummy driver make no testsuite progress report.
+ grep ': .*\.test' stdout && Exit 1
+ # Nor it writes to the log files.
+ test -s foo.log && Exit 1
+ test -s bar.log && Exit 1
+ return $st
+}
+
+# This must be different from the one defined in `test/defs', as that
+# assumes that the driver does proper testsuite progress reporting.
+count_test_results ()
+{
+ total=ERR pass=ERR fail=ERR xpass=ERR xfail=ERR skip=ERR error=ERR
+ eval "$@"
+ st=0
+ grep "^# TOTAL: *$total$" stdout || rc=1
+ grep "^# PASS: *$pass$" stdout || rc=1
+ grep "^# XFAIL: *$xfail$" stdout || rc=1
+ grep "^# SKIP: *$skip$" stdout || rc=1
+ grep "^# FAIL: *$fail$" stdout || rc=1
+ grep "^# XPASS: *$xpass$" stdout || rc=1
+ grep "^# ERROR: *$error$" stdout || rc=1
+ test $st -eq 0 || Exit 1
+}
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+# Basic checks. Also that that "old-style" directives with format
+# "RESULT: test-name" are be ignored now.
+
+cat > foo.test <<END
+FAIL: foo.test
+:test-result: PASS
+END
+echo ERROR: bar.test > bar.test
+mk_check
+count_test_results total=1 pass=1 fail=0 xpass=0 xfail=0 skip=0 error=0
+
+cat > foo.test <<END
+PASS: foo.test
+:test-result: FAIL
+END
+echo SKIP: bar.test > bar.test
+: > bar.test
+mk_check && Exit 1
+count_test_results total=1 pass=0 fail=1 xpass=0 xfail=0 skip=0 error=0
+
+cat > foo.test <<END
+PASS: foo.test
+:test-result: FAIL
+END
+cat > bar.test <<END
+ERROR: foo.test
+:test-result: SKIP
+END
+mk_check && Exit 1
+count_test_results total=2 pass=0 fail=1 xpass=0 xfail=0 skip=1 error=0
+
+cat > foo.test <<END
+:test-result: XFAIL
+:test-result: PASS
+:test-result: SKIP
+END
+cat > bar.test <<END
+:test-result: SKIP
+:test-result: PASS
+:test-result: SKIP
+:test-result: PASS
+:test-result: PASS
+END
+mk_check
+count_test_results total=8 pass=4 fail=0 xpass=0 xfail=1 skip=3 error=0
+
+# Check that all results expected to be supported are *really* supported.
+
+cat > foo.test <<END
+:test-result: PASS
+:test-result: SKIP
+:test-result: XFAIL
+:test-result: FAIL
+:test-result: XPASS
+:test-result: ERROR
+END
+: > bar.test
+mk_check && Exit 1
+count_test_results total=6 pass=1 fail=1 xpass=1 xfail=1 skip=1 error=1
+
+cp foo.test bar.test
+mk_check && Exit 1
+count_test_results total=12 pass=2 fail=2 xpass=2 xfail=2 skip=2 error=2
+
+# Check that we are liberal w.r.t. whitespace use.
+
+: > foo.test
+: > bar.test
+for RESULT in PASS FAIL XPASS XFAIL SKIP ERROR; do
+ sed -e 's/^ *//' -e 's/|//g' >> foo.test <<END
+ |:test-result:$RESULT|
+ |:test-result: $tab $RESULT|
+ |:test-result:$RESULT $tab|
+ |:test-result:$tab$tab $RESULT$tab $tab |
+END
+ echo " $tab $tab$tab :test-result: $RESULT" >> bar.test
+done
+cat foo.test # For debugging.
+cat bar.test # Likewise.
+mk_check && Exit 1
+count_test_results total=30 pass=5 fail=5 xpass=5 xfail=5 skip=5 error=5
+
+:
diff --git a/tests/test-missing.test b/tests/test-missing.test
new file mode 100755
index 0000000..a4ac179
--- /dev/null
+++ b/tests/test-missing.test
@@ -0,0 +1,72 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# parallel-tests:
+# - non-existent scripts listed in TESTS get diagnosed
+# See also related test 'test-missing2.test'.
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cat >> configure.in << 'END'
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TESTS = ok.test zardoz.test
+TEST_LOG_COMPILER = true
+END
+
+: > ok.test
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE -a
+
+./configure
+
+$MAKE check >output 2>&1 && { cat output; Exit 1; }
+cat output
+test -f ok.log
+grep '^PASS: ok\.test' output
+$FGREP 'zardoz.log' output
+test ! -f test-suite.log
+
+TESTS='zardoz2.test' $MAKE -e check >output 2>&1 \
+ && { cat output; Exit 1; }
+cat output
+$FGREP 'zardoz2.log' output
+test ! -f test-suite.log
+
+TEST_LOGS='zardoz3.log' $MAKE -e check >output 2>&1 \
+ && { cat output; Exit 1; }
+cat output
+$FGREP 'zardoz3.log' output
+test ! -f test-suite.log
+
+# The errors should persist even after `test-suite.log'
+# has been created.
+
+: > zardoz.test
+$MAKE check
+rm -f zardoz.test
+
+$MAKE check >output 2>&1 && { cat output; Exit 1; }
+cat output
+$FGREP 'zardoz.log' output
+test ! -f test-suite.log
+
+:
diff --git a/tests/parallel-tests-unreadable-log.test b/tests/test-missing2.test
similarity index 56%
rename from tests/parallel-tests-unreadable-log.test
rename to tests/test-missing2.test
index e8e101b..bc2979a 100755
--- a/tests/parallel-tests-unreadable-log.test
+++ b/tests/test-missing2.test
@@ -14,7 +14,10 @@
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
-# Check that the testsuite driver copes well with unreadable test logs.
+# parallel-tests:
+# - non-existent scripts listed in TESTS get diagnosed, even when
+# all the $(TEST_LOGS) have a dummy dependency.
+# See also related test 'test-missing.test'.
parallel_tests=yes
. ./defs || Exit 1
@@ -24,37 +27,28 @@ AC_OUTPUT
END
cat > Makefile.am << 'END'
-TESTS = foo.test bar.test
-XFAIL_TESTS = bar.test
+TESTS = foobar1.test foobar2.test
+$(TEST_LOGS):
END
-cat > foo.test << 'END'
-#! /bin/sh
-exit 0
-END
-cat > bar.test << 'END'
-#! /bin/sh
-exit 1
-END
-chmod a+x foo.test bar.test
-
$ACLOCAL
$AUTOCONF
$AUTOMAKE -a
-# The testsuite driver will use this variable, so ensure it sanitizes
-# it and do not allow in spurious values from the environment.
-line=PASS; export line
-
./configure
-$MAKE foo.log
-$MAKE bar.log
-chmod a-r foo.log bar.log
-test ! -r foo.log || Exit 77
-$MAKE test-suite.log >stdout && { cat stdout; Exit 1; }
-cat stdout
-grep '^# ERROR: *2$' stdout
-grep '^# ERROR: *2$' test-suite.log
+$MAKE foobar1.log foobar2.log || Exit 99
+test ! -f foobar1.log || Exit 99
+test ! -f foobar1.trs || Exit 99
+test ! -f foobar2.log || Exit 99
+test ! -f foobar2.trs || Exit 99
+
+$MAKE check >output 2>&1 && { cat output; Exit 1; }
+cat output
+grep 'test-suite\.log.*foobar1\.log' output
+grep 'test-suite\.log.*foobar1\.trs' output
+grep 'test-suite\.log.*foobar2\.log' output
+grep 'test-suite\.log.*foobar2\.trs' output
+test ! -f test-suite.log
:
diff --git a/tests/test-trs-basic.test b/tests/test-trs-basic.test
new file mode 100755
index 0000000..33367d3
--- /dev/null
+++ b/tests/test-trs-basic.test
@@ -0,0 +1,138 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# Check parallel harness features:
+# - creation and removal of `.trs' auxiliary files
+# - check some internals regarding the use of `.trs' files.
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cat >> configure.in <<END
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_EXTENSIONS = .sh .test
+TESTS = foo.test bar.sh sub/zardoz.test
+TEST_LOG_COMPILER = $(SHELL)
+SH_LOG_COMPILER = $(SHELL)
+
+## Used to check some internal details.
+tb:
+ echo $(am__TEST_BASES) > $@
+END
+
+mkdir sub
+echo 'exit $FOO_STATUS' > foo.test
+: > bar.sh
+: > sub/zardoz.test
+
+FOO_STATUS=0; export FOO_STATUS
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE -a
+
+./configure
+
+#
+# Check some internal details first.
+#
+
+$MAKE tb
+test x"`cat tb`" = x"foo bar sub/zardoz"
+rm -f tb
+TESTS='foo.test foo2.sh foo3.exe foo4' $MAKE -e tb
+test x"`cat tb`" = x"foo foo2 foo3.exe foo4"
+rm -f tb
+
+#
+# The `test-suite.stamp' file and the `.trs' files get created by
+# "make check" and removed by "make clean" and "make mostlyclean".
+#
+
+: > unrelated.trs
+: > sub/foo.trs
+
+$MAKE check
+test -f foo.trs
+test -f bar.trs
+test -f sub/zardoz.trs
+$MAKE clean
+test ! -f foo.trs
+test ! -f bar.trs
+test ! -f sub/zardoz.trs
+# Unrelated `.trs' files shouldn't be removed.
+test -f unrelated.trs
+test -f sub/foo.trs
+
+# The files should be properly created in case of testsuite failure too.
+FOO_STATUS=1 $MAKE check && Exit 1
+test -f foo.trs
+test -f bar.trs
+test -f sub/zardoz.trs
+$MAKE mostlyclean
+test ! -f foo.trs
+test ! -f bar.trs
+test ! -f sub/zardoz.trs
+# Unrelated `.trs' files shouldn't be removed.
+test -f unrelated.trs
+test -f sub/foo.trs
+
+#
+# Try with a subset of TESTS.
+#
+
+TESTS=foo.test $MAKE -e check
+test -f foo.trs
+test ! -f bar.trs
+test ! -f sub/zardoz.trs
+$MAKE clean
+test ! -f foo.trs
+TESTS='foo.test bar.sh' $MAKE -e check
+test -f foo.trs
+test -f bar.trs
+test ! -f sub/zardoz.trs
+# "make clean" shouldn't remove `.trs' files for tests not in $(TESTS).
+TESTS=bar.sh $MAKE -e clean
+test -f foo.trs
+test ! -f bar.trs
+
+$MAKE clean
+
+#
+# Try with a subset of TEST_LOGS.
+#
+
+TEST_LOGS=sub/zardoz.log $MAKE -e check
+test ! -f foo.trs
+test ! -f bar.trs
+test -f sub/zardoz.trs
+$MAKE clean
+test ! -f sub/zardoz.trs
+TEST_LOGS='foo.log bar.log' $MAKE -e check
+test -f foo.trs
+test -f bar.trs
+test ! -f sub/zardoz.trs
+# "make clean" shouldn't remove `.trs' files for tests whose log
+# is not in $(TEST_LOGS).
+TEST_LOGS=foo.log $MAKE -e clean
+test ! -f foo.trs
+test -f bar.trs
+test ! -f sub/zardoz.trs
+
+:
diff --git a/tests/test-trs-recover.test b/tests/test-trs-recover.test
new file mode 100755
index 0000000..7fff3ae
--- /dev/null
+++ b/tests/test-trs-recover.test
@@ -0,0 +1,166 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# Check parallel harness features:
+# - recovery from deleted `.trs' files, in various scenarios
+# This test is complex and tricky, but that's acceptable since we are
+# testing semantics that are potentially complex and tricky.
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cat >> configure.in <<END
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TESTS = foo.test bar.test baz.test
+TEST_LOG_COMPILER = $(SHELL)
+END
+
+echo 'exit $TEST_STATUS' > foo.test
+echo 'exit $TEST_STATUS' > bar.test
+: > baz.test
+
+TEST_STATUS=0; export TEST_STATUS
+
+# Slower and possible overkill in some situations, but also clearer
+# and safer.
+update_stamp () { $sleep && touch stamp && $sleep; }
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE -a
+
+./configure
+
+: Create the required log files.
+$MAKE check
+
+: Recreate by hand.
+rm -f foo.trs bar.trs baz.trs
+$MAKE foo.trs
+test -f foo.trs
+test ! -f bar.trs
+test ! -f baz.trs
+
+: Recreate by hand, several at the same time.
+rm -f foo.trs bar.trs baz.trs
+$MAKE foo.trs bar.trs
+test -f foo.trs
+test -f bar.trs
+test ! -f baz.trs
+
+: Recreate by hand, with a failing test.
+rm -f foo.trs bar.trs
+TEST_STATUS=1 $MAKE bar.trs baz.trs >stdout || { cat stdout; Exit 1; }
+cat stdout
+test ! -f foo.trs
+test -f bar.trs
+test -f baz.trs
+grep '^FAIL: bar\.test' stdout
+$EGREP '^(baz|foo)\.test' stdout && Exit 1
+
+: Recreate with a sweeping "make check", and ensure that also up-to-date
+: '.trs' files are remade.
+update_stamp
+rm -f foo.trs bar.trs
+$MAKE check
+test -f foo.trs
+test -f bar.trs
+is_newest baz.trs stamp
+
+: Recreate with a sweeping "make check" with failing tests. Again,
+: ensure that also up-to-date '.trs' files are remade -- this time we
+: grep the "make check" output verify that.
+rm -f foo.trs bar.trs
+TEST_STATUS=1 $MAKE check >stdout && { cat stdout; Exit 1; }
+test -f foo.trs
+test -f bar.trs
+grep '^FAIL: foo\.test' stdout
+grep '^FAIL: bar\.test' stdout
+grep '^PASS: baz\.test' stdout
+
+: Recreate with a "make check" with redefined TESTS.
+rm -f foo.trs bar.trs baz.trs
+TESTS=foo.test $MAKE -e check
+test -f foo.trs
+test ! -f bar.trs
+test ! -f baz.trs
+
+: Recreate with a "make check" with redefined TEST_LOGS.
+rm -f foo.trs bar.trs baz.trs
+TEST_LOGS=bar.log $MAKE -e check
+test ! -f foo.trs
+test -f bar.trs
+test ! -f baz.trs
+
+: Interactions with "make recheck" are OK.
+rm -f foo.trs bar.trs baz.log baz.trs
+$MAKE recheck >stdout || { cat stdout; Exit 1; }
+cat stdout
+test -f foo.trs
+test -f bar.trs
+test ! -f baz.trs
+test ! -f baz.log
+grep '^PASS: foo\.test' stdout
+grep '^PASS: bar\.test' stdout
+grep 'baz\.test' stdout && Exit 1
+count_test_results total=2 pass=2 fail=0 xpass=0 xfail=0 skip=0 error=0
+
+: Setup for the next check.
+$MAKE check
+test -f foo.trs
+test -f bar.trs
+test -f baz.trs
+
+: Recreate by remaking the global test log, and ensure that up-to-date
+: '.trs' files are *not* remade.
+update_stamp
+rm -f foo.trs bar.trs test-suite.log
+$MAKE test-suite.log >stdout || { cat stdout; Exit 1; }
+cat stdout
+grep '^PASS: foo\.test' stdout
+grep '^PASS: bar\.test' stdout
+grep 'baz\.test' stdout && Exit 1
+stat *.trs *.log stamp || : # For debugging.
+# Check that make has updated what it needed to, but no more.
+test -f foo.trs
+test -f bar.trs
+is_newest stamp baz.trs
+is_newest test-suite.log foo.trs bar.trs
+
+: Setup for the next check.
+$MAKE check
+test -f foo.trs
+test -f bar.trs
+test -f baz.trs
+
+: Interactions with lazy test reruns are OK.
+rm -f foo.trs
+update_stamp
+touch bar.test
+RECHECK_LOGS= $MAKE -e check >stdout || { cat stdout; Exit 1; }
+cat stdout
+# Check that make has updated what it needed to, but no more.
+test -f foo.trs
+is_newest bar.trs bar.test
+is_newest stamp baz.trs
+grep '^PASS: foo\.test' stdout
+grep '^PASS: bar\.test' stdout
+grep 'baz\.test' stdout && Exit 1
+
+:
diff --git a/tests/test-trs-recover2.test b/tests/test-trs-recover2.test
new file mode 100755
index 0000000..bc94b3b
--- /dev/null
+++ b/tests/test-trs-recover2.test
@@ -0,0 +1,133 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# Check parallel harness features:
+# - recovery from unreadable `.trs' files, in various scenarios
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cat >> configure.in <<END
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TESTS = foo.test bar.test
+TEST_LOG_COMPILER = true
+END
+
+: > foo.test
+: > bar.test
+
+TEST_STATUS=0; export TEST_STATUS
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE -a
+
+./configure
+
+: > t
+chmod a-r t
+test ! -r t || Exit 77
+rm -f t
+
+: Create the required log files.
+$MAKE check
+test -f foo.trs
+test -f bar.trs
+
+: Recreate with a sweeping "make check".
+chmod a-r bar.trs
+$MAKE check
+test -f foo.trs
+test -r foo.trs
+test -f bar.trs
+test -r bar.trs
+
+: Again, this time with one .trs file missing and the other
+: one unreadable.
+rm -f foo.trs
+chmod a-r bar.trs
+$MAKE check
+test -f foo.trs
+test -r foo.trs
+test -f bar.trs
+test -r bar.trs
+
+: Again, but using "make recheck" this time.
+rm -f foo.trs
+chmod a-r bar.trs
+$MAKE recheck >stdout || { cat stdout; Exit 1; }
+cat stdout
+test -f foo.trs
+test -r foo.trs
+test -f bar.trs
+test -r bar.trs
+grep '^PASS: foo\.test' stdout
+grep '^PASS: bar\.test' stdout
+
+: More complex interactions with "make recheck" are OK.
+chmod a-r bar.log bar.trs
+$MAKE recheck >stdout || { cat stdout; Exit 1; }
+cat stdout
+test -f bar.trs
+test -r bar.trs
+grep '^PASS: bar\.test' stdout
+grep 'foo\.test' stdout && Exit 1
+count_test_results total=1 pass=1 fail=0 xpass=0 xfail=0 skip=0 error=0
+
+: Recreate by remaking the global test log.
+chmod a-r foo.trs
+rm -f test-suite.log
+$MAKE test-suite.log >stdout || { cat stdout; Exit 1; }
+cat stdout
+test -f foo.trs
+test -r foo.trs
+grep '^PASS: foo\.test' stdout
+grep 'bar\.test' stdout && Exit 1
+# Also test that have only run before should be counted in the final
+# testsuite summary.
+grep '^# TOTAL: *2$' stdout
+
+: Setup for the next check.
+: > baz.test
+sed 's/^TESTS =.*/& baz.test/' Makefile > t
+diff t Makefile && Exit 99
+mv -f t Makefile
+$MAKE check
+test -f foo.trs
+test -f bar.trs
+test -f baz.trs
+
+: Interactions with lazy test reruns are OK.
+chmod a-r foo.trs
+$sleep
+touch stamp
+$sleep
+touch bar.test
+RECHECK_LOGS= $MAKE -e check >stdout || { cat stdout; Exit 1; }
+cat stdout
+test -r foo.trs
+is_newest bar.trs bar.test
+grep '^PASS: foo\.test' stdout
+grep '^PASS: bar\.test' stdout
+grep 'baz\.test' stdout && Exit 1
+# Also test that have only run before should be counted in the final
+# testsuite summary.
+grep '^# TOTAL: *3$' stdout
+
+:
diff --git a/tests/trivial-test-driver b/tests/trivial-test-driver
index d334a93..ab73d8d 100644
--- a/tests/trivial-test-driver
+++ b/tests/trivial-test-driver
@@ -36,10 +36,12 @@ set -u
test_name=INVALID.NAME
log_file=BAD.LOG
+trs_file=BAD.TRS
while test $# -gt 0; do
case $1 in
--test-name) test_name=$2; shift;;
--log-file) log_file=$2; shift;;
+ --result-file) trs_file=$2; shift;;
# Ignored.
--expect-failure) shift;;
--color-tests) shift;;
@@ -52,16 +54,18 @@ while test $# -gt 0; do
shift
done
-## Run the test script, get test cases results, display them on console.
+## Log file header.
+{
+ echo "RUN: $test_name"
+ echo "RUN: $test_name" | sed 's/./=/g'
+ echo
+} > $log_file
-tmp_output=$log_file-output.tmp
-tmp_results=$log_file-results.tmp
-tmp_status=$log_file-status.tmp
+## Run the test script, get test cases results, display them on console.
-"$@" 2>&1 | tee $tmp_output | (
+"$@" 2>&1 | tee -a $log_file | (
i=0 st=0
- exec 5> $tmp_results
- : > $tmp_status
+ exec 5> $trs_file
while read line; do
result=
case $line in
@@ -81,38 +85,18 @@ tmp_status=$log_file-status.tmp
echo >&5
fi
done
- test $st -eq 0 || echo fail > $tmp_status
+ if test $st -eq 0; then
+ recheck=no
+ copy_in_global_log=no
+ else
+ recheck=yes
+ copy_in_global_log=yes
+ fi
+ echo ":recheck: $recheck" >&5
+ echo ":copy-in-global-log: $copy_in_global_log" >&5
+ exec 5>&-
) | awk '{ print $0 ", testcase " NR }'
-if test ! -s $tmp_status; then
- global_result=PASS
- recheck=no
- copy_in_global_log=no
-else
- global_result=FAIL
- recheck=yes
- copy_in_global_log=yes
-fi
-
-## Write the log file.
-
-{
- echo "$global_result: $test_name"
- echo "$global_result: $test_name" | sed 's/./=/g'
- # Blank lines required for having the output be valid RST.
- echo
- echo ":recheck: $recheck"
- echo
- echo ":copy-in-global-log: $copy_in_global_log"
- echo
- cat $tmp_results
- echo
- echo --------------------
- echo
- cat $tmp_output
-} > $log_file
-rm -f $tmp_output $tmp_results $tmp_status
-
## And we're done.
exit 0
--
1.7.2.3
- [GSoC] Some patches for testsuite harness improvements and TAP support introduction, Stefano Lattarini, 2011/07/14
- [PATCH 1/5] {test-protocols} parallel-tests: make parsing of test results safer, Stefano Lattarini, 2011/07/14
- Re: [PATCH 1/5] {test-protocols} parallel-tests: make parsing of test results safer, Ralf Wildenhues, 2011/07/18
- Re: [PATCH 1/5] {test-protocols} parallel-tests: make parsing of test results safer, Stefano Lattarini, 2011/07/19
- Re: [PATCH 1/5] {test-protocols} parallel-tests: make parsing of test results safer, Ralf Wildenhues, 2011/07/19
- Re: [PATCH 1/5] {test-protocols} parallel-tests: make parsing of test results safer, Stefano Lattarini, 2011/07/20
- Re: [PATCH 1/5] {test-protocols} parallel-tests: make parsing of test results safer, Ralf Wildenhues, 2011/07/20
- Re: [PATCH 1/5] {test-protocols} parallel-tests: make parsing of test results safer, Stefano Lattarini, 2011/07/21
- [PATCH 1/2] test harness: allow more metadata in log files, Stefano Lattarini, 2011/07/29
- Re: [PATCH 1/2] test harness: allow more metadata in log files,
Stefano Lattarini <=
[PATCH 2/5] {test-protocols} parallel-tests: new recognized test result 'ERROR', Stefano Lattarini, 2011/07/14
[PATCH 3/5] {test-protocols} parallel-tests: simplify testsuite summary, Stefano Lattarini, 2011/07/14
[PATCH 4/5] {test-protocols} tests defs: new auxiliary function 'count_test_results', Stefano Lattarini, 2011/07/14
[PATCH 5/5] {test-protocols} tap: add experimental TAP-aware driver, Stefano Lattarini, 2011/07/14