Re: parallel-tests: redo lazy checking: recheck and RECHECK_LOGS.

[Akim Demaille]

Le 29 mars 09 à 00:04, Ralf Wildenhues a écrit :

> Hi Akim,

Hi Ralf!

> Agreed.  While I won't change the TESTS=foo.test semantics, we can
> publish that the user can use TEST_LOGS=foo.log to limit the tests  
> to be
> run.

Never thought about that :)  Thanks.


> > - LAZY_TEST_SUITE is not flexible enough
> >  It is global, it should be per test-case.
>
> Indeed.  I like your idea, but I don't like the naming scheme too  
> much,
> nor do I think we need to be backwards compatible with an unpublished
> interface.  I'm removing the LAZY_TEST_SUITE API and replacing it  
> with a
> RECHECK_LOGS API (instead of your STRICT_TESTS).
>
> Sorry if this causes trouble for you.

Not sure that there's an Eclipse component to rename Make macros, but  
I can afford a query-replace :)



> > I often use
> >
> >         STRICT_TEST_LOGS = $(shell $(LIST_FAILED_TEST_LOGS))
> >
> > which makes all failing test strict.  In other words, successful  
> > tests
> > are not rerun by "make check", but failing tests are.  This is  
> > because
> > our test suite sometimes hits the limit of the machine, and tests
> > timeout for no sound reason.  So there is no reason to rerun  
> > successful
> > tests, but failing test might pass if the machine has a lesser load.
>
> This is what the 'recheck' target does, right?

Not exactly.  make recheck runs only the tests that failed, while  
here, in addition, the tests whose dependencies have changed will be  
run again.  Not to mention test that where not run at all.


> One thing that one needs to look out for is, when overriding variables
> in recursive `make' instances is that non-GNU make don't override by
> default unless you use `make -e' and pass via the environment.
> Requiring our users to do so if they want to override variables is ok
> IMVHO, because then they are warned that their environment needs to be
> clean.  OTOH using -e internally is not ok at all, as the users may  
> not
> be aware, and their environment could cause subtle breakage.  So
> consequently we can only transport overrides one recursion deep and  
> not
> any further, unless each lower re-sets the variables on the make  
> command
> line.

Wow, man, you *are* an expert...

I have not understood exactly what you are referring too, but why  
don't we just pass TESTS=$(TESTS) explicitly?  Or ask the users to add  
his "precious macros" to AM_MAKEFLAGS?

> This detail was BTW the reason that I have backed off of allowing  
> shell
> globbing for TESTS or TEST_LOGS.  It is just too complicated to get
> right portably, causes other problems (e.g., I do not want to have a
> rule that has both TESTS and TEST_LOGS expanded in one shell command:
> it might overrun command line length limits), and can be emulated by  
> the
> user on the command line or with a script (or, for GNU make, a simple
> wrapper target).  Thus this patch documents such an example.

Can't we have it when using GNU Make?  Can't we have some sort of  
#ifdef HAVE_GNU_MAKE sections in the output Makefile?  I'm not  
referring to AM_CONDITIONAL, but to some nice run-time trick.



> +## Rerun all FAILed or XPASSed tests (as well as all whose logs are  
> out
> +## of date or do not exist).
> +recheck-TESTS:
> +       @list='$(TEST_LOGS)';                                           \
> +       logs=`for f in $$list; do                                       \
> +               if read line < $$f; then                             \
> +                 case $$line in FAIL*|XPASS*) echo $$f;; esac;         \
> +               else echo $$f; fi;                                      \
> +             done | tr '\012\015' '  '`;                               \
> +       $(MAKE) $(AM_MAKEFLAGS) check-TESTS RECHECK_LOGS="$$logs"

Maybe on purpose, maybe not, you dropped recheck-html.


This is great!