Re: [PATCH 2/5] {test-protocols} parallel-tests: new recognized test result 'ERROR'

[Ralf Wildenhues]

* Stefano Lattarini wrote on Fri, Jul 15, 2011 at 12:37:07AM CEST:
> * lib/am/check.am ($(TEST_SUITE_LOG)): Recognize a new test result
> `ERROR'.  Use it when encountering unreadable test logs (previously
> a simple `FAIL' was used in this situations).
> * lib/test-driver: Set the global test result to `ERROR' when the
> test exit status is 99.  When doing colorized output, color `ERROR'
> results in magenta.
> * doc/automake.texi (Log files generation and test results
> recording): Update by also listing `ERROR' among the list of valid
> `:test-results:' arguments.
> * NEWS: Update.
> * tests/trivial-test-driver: Update.
> * tests/parallel-tests.test: Likewise.
> * tests/parallel-tests-harderror.test: Likewise.
> * tests/parallel-tests-no-spurious-summary.test: Likewise.
> * tests/test-driver-global-log.test: Likewise.
> * tests/test-driver-recheck.test: Likewise.
> * tests/test-driver-custom-multitest-recheck.test: Likewise.
> * tests/test-driver-custom-multitest-recheck2.test: Likewise.
> * tests/test-driver-custom-multitest.test: Likewise.
> * tests/test-driver-custom-no-html.test: Likewise.
> * tests/test-driver-end-test-results.test: Likewise.
> * tests/color.test: Likewise.  Also, make stricter, and also test
> from VPATH.
> * tests/color2.test: Likewise, and improve syncing with color.test.
> * tests/parallel-tests-exit-statuses.test: New test.
> * tests/parallel-tests-console-output.test: Likewise.
> * tests/Makefile.am (TESTS): Update.

> --- a/NEWS
> +++ b/NEWS
> @@ -13,6 +13,10 @@ New in 1.11a:
>  
>  * Changes to Automake-generated testsuite harnesses:
>  
> +  - The parallel-tests harness has a new test result 'ERROR', that can be
> +    used to signal e.g., unexpected or internal errors, or failures to set
> +    up test case scenarios.

The NEWS entry could be a bit more precise in that the ERROR state was
actually used before already with exit status 99, just that it is named
ERROR now.  (Please check whether the semantics of 99 already were in a
stable release, otherwise this NEWS entry could of course advertise it
as new.)

A description of ERROR semantics needs to be part of automake.texi as
well.

Also, are you going to followup with Autoconf to rename Autotest's hard
failure state ERROR as well?  We should agree on common naming and
semantics, so the two systems are not harder to learn than necessary.

I'm OK with this change once these issues are resolved.

As a minor detail however, please remove the '====' greps, I've already
mentioned why I consider '=' in the output not a good idea.

Thanks,
Ralf

> --- a/doc/automake.texi
> +++ b/doc/automake.texi
> @@ -9248,10 +9248,10 @@ leading whitespace will not be ignored.
>  
>  @c Keep this in sync with lib/am/check-am:$(TEST_SUITE_LOG).
>  The only recognized test results are currently @code{PASS}, @code{XFAIL},
> address@hidden, @code{FAIL} and @code{XPASS}.  These results, when declared
> -with @code{:test-result:}, can be optionally followed by text holding
> -the name and/or a brief description of the corresponding test; the
> address@hidden harness will ignore such extra text when
> address@hidden, @code{FAIL}, @code{XPASS} and @code{ERROR}.  These results,
> +when declared with @code{:test-result:}, can be optionally followed by
> +text holding the name and/or a brief description of the corresponding
> +test; the @option{parallel-tests} harness will ignore such extra text when
>  generating @file{test-suite.log} and preparing the testsuite summary.
>  Also, @code{:test-result:} can be used with a special ``pseudo-result''
>  @code{END}, that will instruct the testsuite harness to stop scanning