Re: parallel-tests: redo lazy checking: recheck and RECHECK_LOGS.

[Akim Demaille]

Le 31 mars 09 à 01:01, Ralf Wildenhues a écrit :

Salut Ralf !

> * Akim Demaille wrote on Mon, Mar 30, 2009 at 09:50:59PM CEST:
> > Le 29 mars 09 à 00:04, Ralf Wildenhues a écrit :
> >
> > > > I often use
> > > >
> > > >         STRICT_TEST_LOGS = $(shell $(LIST_FAILED_TEST_LOGS))
> > > >
> > > > which makes all failing test strict.  In other words, successful
> > > > tests are not rerun by "make check", but failing tests are.  This  
> > > > is
> > > > because our test suite sometimes hits the limit of the machine, and
> > > > tests timeout for no sound reason.  So there is no reason to rerun
> > > > successful tests, but failing test might pass if the machine has a
> > > > lesser load.
> > >
> > > This is what the 'recheck' target does, right?
> >
> > Not exactly.  make recheck runs only the tests that failed, while  
> > here,
> > in addition, the tests whose dependencies have changed will be run  
> > again.
> > Not to mention test that where not run at all.
>
> I don't think your interpretation is right.  Those tests whose
> dependencies have changed will be rerun in any case, with your
> recheck, with my recheck, with above.  You cannot prevent it unless
> you remove those dependency statements from the makefile.

Err, really?  My code reads:

# Get the list of tests that failed.
LIST_FAILED_TEST_LOGS =                                                 \
  perl -e 'print join " ",                                            \
                 grep {open(_) && <_> =~ /^(FAIL|XPASS): /} (@ARGV);'     \
       -- $(TEST_LOGS)
.PHONY: recheck recheck-html
recheck recheck-html:
        @target=$$(echo $@ | sed -e 's/^re//');                 \
        if test -f $(TEST_SUITE_LOG); then                      \
          TESTS=$$($(LIST_FAILED_TEST_LOGS));                   \
          $(MAKE) $(AM_MAKEFLAGS) $$target TESTS="$$TESTS";   \
        else                                                    \
          $(MAKE) $(AM_MAKEFLAGS) $$target;                     \
        fi

So I really get the list of logs that were FAIL'd, remove them, and  
run them as TESTS.  Here I'm using the fact that TESTS accepts *.log  
in my implementation.  I don't see why logs that passed but are  
obsoleted would run again.


> > I have not understood exactly what you are referring too, but why  
> > don't
> > we just pass TESTS=$(TESTS) explicitly?
>
> Because it can cause command line length issues.  Because variable
> overriding is inherently fragile: it can subtly change the meaning
> of the underlying makefile.
>
> When another indirection for globbing were added, we'd have to pass
> $(TESTS) through more than one $(MAKE).  And we'd have to pass
> $(TEST_LOGS) as well, if we were to allow globbing on that.  And
> I would like to avoid passing both, that halves our available length
> limit.

Wow.  What's the worse case for this length?  How about having another  
mode to run the test suite, based on a temporary MANITEST file with  
the list of files to run?  Maybe a __am_tests.mk file and a 'make -f  
__am_tests.mk check'?  It would have list the dependencies from test- 
suite.log to the individual logs, and include the main Makefile.  I  
dunno, maybe I don't make any sense here, or maybe there are  
complications I have no foreseen.

Are there non-GNU Makes that support -j?

> > Or ask the users to add his "precious macros" to AM_MAKEFLAGS?
>
> We could do that.  In fact, I intend to add a note to this extent to  
> the
> Autoconf manual section about `make' portability, but AM_MAKEFLAGS
> should be mentioned in automake.texi too I guess.  What I don't know  
> is
> whether it was intended as user variable or as makefile.am author  
> realm.

Given it's name, I think it should be userland.  __AM_ whatever is a  
different beast, but AM_CFLAGS and so forth are definitely userland.


> > Can't we have it when using GNU Make?  Can't we have some sort of  
> > #ifdef
> > HAVE_GNU_MAKE sections in the output Makefile?  I'm not referring to
> > AM_CONDITIONAL, but to some nice run-time trick.
>
> Produce a GNUmakefile in the build directory.  Let this GNUmakefile
> include the Makefile.  Put all your nice goodies in GNUmakefile.
>
> An example of this is what the gnulib module gnumakefile does.
> For example, it allows you to
>  make -j3 clean all check distcheck

nice :)

> and it will work!  In a git repo, and with the git .version strategy
> used by several gnulib-using packages, it will autoreconf so the  
> tarball
> will have an up to date version string.

Yes, I also have some GNUmakefiles where I want to play that kind of  
trick.  But this kind of clashes with users that already do.  I was  
looking for something for Automake, not for the Automake user.