Re: DejaGnu unit testing protocol

[David Malcolm]

On Wed, 2020-07-22 at 17:05 -0500, Jacob Bachmeyer wrote:
> I had been planning to look at this later, but fixing PR42399
> brought 
> this front-and-center.  The DejaGnu unit testing protocol is not
> (yet) 
> documented (will be fixed as part of PR42399) and there is a strange 
> quirk in its handler.
> 
> What produces lines with prefixes matching {[0-9][0-9]:..:..:}
> and/or 
> why is host_execute looking for that pattern?  These lines have the 
> effect only of resetting a timeout counter and printing the line if 
> verbose is 3 or higher but are otherwise ignored.
> 
> 
> The DejaGnu unit testing protocol is very simple, and I think I can
> work 
> with this fairly well for the long-term.  DejaGnu spawns a unit test 
> program and reads that program's output.  Arguments for the unit
> test 
> program can be specified in the testsuite.  The unit test program
> can 
> produce any output, but DejaGnu is sensitive to lines beginning with
> a 
> tab character, followed by uppercase letters, followed by a colon.  
> These lines (currently) indicate test results and (in the future) 
> provide additional information to effect proper testing.  (Examples
> of 
> future extensions now planned include support for known failures and 
> framework-assessed expected failures.  Currently, the dejagnu.h
> header 
> provides xfail/xpass functions to allow unit tests to declare
> expected 
> failure, but host_execute does not recognize these results; this
> second 
> bug is now slated to be fixed for 1.6.3.)
> 
> Unit test programs may also produce other output for the benefit of
> a 
> developer directly running them or reading the DejaGnu log, but the 
> sequence {\n\t[][[:upper:]]*:} should be considered reserved for the 
> DejaGnu unit testing support.
> 
> 
> Anyone know why that pattern was included in host_execute or have
> other 
> comments?

In terms of "other comments":

FWIW within gcc, the jit.dg testsuite for libgccjit has a copy of
host_execute ("fixed_host_execute") to workaround issues I've run into:

https://gcc.gnu.org/git/?p=gcc.git;a=blob;f=gcc/testsuite/jit.dg/jit.exp

It has drifted somewhat from the DejaGnu original; for example it
gained the ability to parse valgrind output and convert leaks into
pass/fail results.

(I also ran into the issue that dejagnu.h's pass/fail C functions
aren't thread-safe, which I hack around in my testsuite, replacing them
in multi-threaded tests with ones guarded by a mutex).

Hope this is constructive
Dave