Re: [PATCH 1/5] {test-protocols} parallel-tests: make parsing of test results safer

[Ralf Wildenhues]

* Stefano Lattarini wrote on Fri, Jul 15, 2011 at 12:36:17AM CEST:
> The new code for parsing the testsuite-generated `.log' files,
> as introduced in commit `v1.11-872-gc96b881', considers each
> `:test-result:' field anywhere in a `.log' file as a declaration
> of a test result, and accounts for it as such in the testsuite
> summary.  Unfortunately this could easily cause spurious test
> failures being reported in the testsuite summary.  This happened
> in practice with the Automake's own testsuite; for example:
> 
>   $ make check TESTS='check12-p.test'; echo exit: $?
>   ...
>   PASS: check12-p.test
>   =====================================
>   4 of 5 tests failed
>   See tests/test-suite.log
>   Please report to address@hidden
>   =====================================
>   make[2]: *** [test-suite.log] Error 1
>   make: *** [check-am] Error 2
>   exit: 2
> 
> This change introduces a new special `:test-result:' "END", that,
> when seen, prevents the rest of the log file from being parsed.
> 
> For more information, refer to the thread:
> <http://lists.gnu.org/archive/html/automake-patches/2011-06/msg00199.html>
> 
> * lib/am/check.am ($(TEST_SUITE_LOG)): Stop the parsing of a log
> file as soon as the special ":test-result:END" directive is seen.
> Related changes and enhancements.
> * lib/test-driver: Protect the rest of the log after the result
> lined with a ":test-result:END" directive.
> * tests/parallel-tests-no-spurious-summary.test: New test.
> * tests/test-driver-end-test-results.test: Likewise.
> * tests/Makefile.am (TESTS): Update.

I'm still not sold on this.  It is not as robust as a test protocol
could be; also I haven't seen this approach being used in any other test
suite environments.  Whether some line is considered having a result or
not depends on (possibly far-away) context, on whether aggregation with
other results has happened.  When passed through email, misquoting can
change not only the interpretation of the misquoted text (which could be
expected) but possibly also of later, correctly quoted (or not quoted)
text.

Please, this is really important: we need to research the other test
protocols, what they do to be robust here.  Don't NIH here, because the
experience we have is not enough to not mess up this.  Consider this
research as part of the work needed for the SoC assignment, it is a very
important part (as it is not easily corrected once released).

Thanks,
Ralf