Re: [PATCH 11] Fix LAZY_TEST_SUITE handling and $(TEST_SUITE_LOG) recreation.

[Akim Demaille]

Le 14 mars 09 à 12:01, Ralf Wildenhues a écrit :

> Oh, what a joyful patch; cost me most brains.  Here's why:
>
> It would have been cool if
>  make check LAZY_TEST_SUITE=yes
>
> on an up to date test directory would have no work to do at all.
> Well.  Couple of things wrong with the prior semantics:
> - there would be no summary output on stdout in that case;
> - the exit status of `make' would not reflect the fact whether any of
>  the tests failed.  I'd consider that a bug.

Hi Ralf,

Thanks for all this wonderful work!  I can't wait to use it!


For the records, here is the version I use currently, some of bugs you  
mentioned being fixed.  I don't know which was the last I sent, so I  
can't send a patch.  Among changes that I believe, are worth  
mentioning are:

- make recheck
  runs only the test that failed last time.

> # Get the list of tests that failed.
> LIST_FAILED_TEST_LOGS =                                                 \
>   perl -e 'print join " ",                                            \
>                  grep {open(_) && <_> =~ /^(FAIL|XPASS): /}  
> (@ARGV);'	\
>        -- $(TEST_LOGS)
> .PHONY: recheck recheck-html
> recheck recheck-html:
>         @target=$$(echo $@ | sed -e 's/^re//');                 \
>         if test -f $(TEST_SUITE_LOG); then                      \
>           TESTS=$$($(LIST_FAILED_TEST_LOGS));                   \
>           $(MAKE) $(AM_MAKEFLAGS) $$target TESTS="$$TESTS";   \
>         else                                                    \
>           $(MAKE) $(AM_MAKEFLAGS) $$target;                     \
>         fi

Contrary to your implementation, I still accept TESTS=foo.log whereas  
the genuine test is foo.test.  This makes many things easier, and it  
is also very convenient for the user.  I have a test suite which uses  
many different extensions, in which case it is simpler for me brains  
to simply enter the (base)name of the test(s).


- LAZY_TEST_SUITE is not flexible enough
  It is global, it should be per test-case.  I initially started with  
LAZY_TESTS being the list of lazy tests, but it was inconvenient  
because being lazy means: do not remove my *.log first.  So actually  
you want the list of strict test logs and remove them.  So it is  
easier to ask the user to report the set of STRICT_TESTS, which  
defaults to $(TESTS). I still use LAZY_TEST_SUITE in check.mk for  
backward compatibility issues, but I would depart from it.

> # Run all the tests.
> check-TESTS:
>         @if test -n '$(LAZY_TEST_SUITE)'; then  \
>           rm -f $(STRICT_TEST_LOGS);            \
>         else                                    \
>           rm -f $(TEST_LOGS);                   \
>         fi
>         @rm -f $(TEST_SUITE_LOG)
>         @$(MAKE) $(TEST_SUITE_LOG)

I often use

        STRICT_TEST_LOGS = $(shell $(LIST_FAILED_TEST_LOGS))

which makes all failing test strict.  In other words, successful tests  
are not rerun by "make check", but failing tests are.  This is because  
our test suite sometimes hits the limit of the machine, and tests  
timeout for no sound reason.  So there is no reason to rerun  
successful tests, but failing test might pass if the machine has a  
lesser load.

check.mk
Description: Binary data