Re: Excluding FAILs from UNSUPPORTED test cases

[Jacob Bachmeyer]

Arsen Arsenović wrote:
> On Monday, 26 September 2022 03:11:43 CEST Jacob Bachmeyer wrote:
>   
> > The tests that it fixed probably need some attention in the form of
> > moving some "preclean" steps to GCC's ${tool}-dg-test.
> >     
> I'll add giving them a look to my TODO.
>   

Please keep me informed about the results of looking into those; if this 
has actually exposed bugs in the GCC testsuite, I may add (as a 
testsuite debugging aid) an option to run ${tool}-dg-prune in an extra 
pass before analyzing the output.  I am tentatively considering a 
variable DG_PRUNE_EARLY_AND_OFTEN that could be set on the runtest 
command line to arrange for this.

> > Keeping the current interface while fixing the {PASS,FAIL}+UNSUPPORTED
> > issue is also possible:  we can build up a list of results from
> > scanning the messages, but not actually report them to the framework
> > until after checking the return value of ${tool}-dg-prune for an
> > "abandon test" flag.  Note that ${tool}-dg-test could produce this
> > flag in lieu of actual output and ${tool}-dg-prune simply pass it
> > along.  (I suspect that the current Tcl list facilities that would
> > enable this were not yet available when dg.exp was written more than
> > 20 years ago.)
> >
> > The main reason I am reluctant to simply accept the "abandon test"
> > flags directly from ${tool}-dg-test is that that would completely
> > break backwards compatibility:  testsuites depending on this would
> > absolutely need DejaGnu 1.6.4, and I think we may be able to get most
> > of the way there (at least getting an UNSUPPORTED result reliably
> > into the summary) with existing code.
> >     
> Right, I didn't think of that.  Indeed, aggregating the results to 
> dispatch them after pruning (but before excess error) should cover all 
> of these cases.
>   

This is still the plan, although I must ask if you know of a reason 
(other than inertia) not to continue aggregating results until the very 
end of the dg-test procedure.

> > Excellent!  It is now tracked as PR58065.  I expect to have a
> > regression test and a preliminary patch in a few days.
> >     
> That's great.  Thank you very much!
>   

My apologies for the delay:  developing unit tests for dg.exp has proven 
to be more involved than initially expected.  (Starting with writing a 
pure-Tcl VFS shim for the internal unit test support infrastructure 
because a getdirs test in utils.test will fail if any more subtrees are 
added to testsuite/runtest.libs.  Why yes, I now plan to eventually also 
make the other tests independent of the actual filesystem now that I 
have had to write this shim.)


-- Jacob