Re: [PATCH] tests init: new subroutine 'fatal_', for hard errors

[Stefano Lattarini]

On Tuesday 07 June 2011, Eric Blake wrote:
> On 06/07/2011 02:22 PM, Bruno Haible wrote:
> > Stefano Lattarini wrote:
> >> +fatal_ () { warn_ "$ME_: hard error: $@"; Exit 99; }
> > 
> > What is a "hard error"?
> > 
> > I thought a test could either succeed, or fail, or produce no result due to
> > unexpected circumstances, and this last case was called "framework failure".
> > Quartum non datur.
> > 
> > So, what is a "hard error"? 
> 
> Exit status 99 is special to automake - it says quit the testsuite now,
> regardless of how many tests remain
>
Well, no, it means "count this outcome as a failure, even if the testcase is
expected to fail" (i.e., in automake, it's listed in $(XFAIL_TESTS)).  To see
a scenario where this can useful (apart the already reported one "the setup
of the test scenario failed"), assume you are testing a C program in alpha
stage; if you are doing test-driven developement, it's acceptable to have
some (or most) of you tests failing; but it's not acceptable for your program
to experience a segfault, not even in a test that is supposed to fail.  In
this case, exiting with status `99' will allow you to see the problem early,
instead of keeping it buried in a "just an XFAIL, all is well" report.

> because this error is indicative of
> something so seriously broken that many other tests will likely fail as
> a knock-on effect of the bug exposed by this failure.
>
While the behaviour you describe might be surely useful in some circumstances,
it's not currently implemented in Automake.  If I'm not mistaken, it was
considered in the design phase of the parallel-tests driver, but not included
in the final implementation.  I can't remember why ATM, I could go digging for
some links tomorrow if you're interested.

> All other failure exit status let automake proceed on with the remainder
> of the testsuite.
> 
> 

Regards,
  Stefano