Re: running tests under in a tests/ directory

[Ralf Wildenhues]

* aaragon wrote on Wed, Jun 03, 2009 at 09:22:59PM CEST:
> Ralf Wildenhues wrote:
> > * aaragon wrote on Sat, May 30, 2009 at 07:34:13PM CEST:
> >> check:        all
> >>       @echo "Running tests"
> >>       cd $(TSTDIR) && $(MAKE) $(AM_MAKEFLAGS) -s test

> >> Except that if I change 'check: all' to something else, then it
> >> doesn't work anymore.
> > 
> > "doesn't work"  is too vague.  What does not work, what error message do
> > you get, more details please?
> 
> In your first reply you suggested to use check-local, but if I put in the
> Makefile.am in the root director:
> 
> check:        check-local

Ah, that's not what I meant, sorry for being imprecise.  I meant to
use

check-local:
        @echo ...

IOW, replace 'check' with 'check-local'.  And then, because 'check' will
invoke 'check-local', and 'check' will recurse already anyway, you can
omit the recursion and just let it invoke the check-am target in
tests/Makefile.

> >> Now, from the code that I got from the internet, I really liked the
> >> fact that you don't need to specify the name of the tests in the
> >> Makefile.am file because of the use of wildcards. Is there a way to
> >> make something similar here that is really portable?
> > 
> > Not really; well, at the very least you will need to have a naming
> > scheme such that all tests have some specified suffix(es), for example
> > '.test'.  Your sources would then match '*.test.cpp'.

> Can you be more specific on this? All my files start with the word 'test',
> but I can change the file names so that they end in 'test.cpp' if needed.
> However, I tried to put 
> 
> TESTEXECS = 'test*.cpp'
> 
> in the Makefile.am under the tests directory and it gives me an error:

Yeah, currently it is not possible to use wild cards there.  Sorry for
giving that false impression, I didn't think it fully through.

> Ralf Wildenhues wrote:
> >> If I make one of the tests fail (by deliberately changing one of the
> >> verified files), I get a message saying that a binary operator is
> >> expected:
> >> 
> >> address@hidden/Documents/workspace/cpputils/tests$make check
> >> make  check-TESTS
> >> FAIL: test001
> >> PASS: testcppblas
> >> /bin/sh: line 0: test: Alejandro: binary operator expected
> >> ========================
> >> 1 of 2 tests failed
> >> See tests/test-suite.log
> >> ========================
> >> make[2]: *** [test-suite.log] Error 1
> >> make[1]: *** [check-TESTS] Error 2
> >> make: *** [check-am] Error 2
> >> 
> >> Is this normal?
> > 
> > Not sure, it could be a bug in my suggested code or in Automake.  Can
> > you post the output of
> > 
> >   make check SHELL=/bin/sh\ -x
> > 
> > for that, as well as the tests/Makefile.am and the automake version
> > you're using?  Thanks.

> I'm using:
> address@hidden/Documents/workspace/cpputils$automake --version
> automake (GNU automake) 1.11

> If I type 'make check' and the tests are successful, I get the following
> error at the end:
> 
> address@hidden/Documents/workspace/cpputils$make check

> -------------------------------------------------------------------
> Running tests
> cd tests && make  -s test
> make[1]: *** No rule to make target `test'.  Stop.
> make: *** [check] Error 2
> address@hidden/Documents/workspace/cpputils$
> 
> 
> I think that error is because of the following line in the Makefile.am in
> the root director:
> 
>       cd $(TSTDIR) && $(MAKE) $(AM_MAKEFLAGS) -s test
> 
> Shouldn't I just use
> 
>       cd $(TSTDIR) && $(MAKE) $(AM_MAKEFLAGS) -s
> 
> ?

Well, if there is no "test" rule defined in tests/Makefile.am, then it
cannot be run.  But automake already provides recursion for you, with
the check rule, so you can just use that and need not build your own.
See above.

> Here I make one of the tests fail deliberately. This is the output of "make
> check SHELL=/bin/sh\ -x" for this case, where you can see the error I was
> referring to in my post at the end:
> 
> address@hidden/Documents/workspace/cpputils$make check SHELL=/bin/sh\ -x
> + test '!' -f config.h

> + msg='1 of 2 tests failed.  See tests/test-suite.log.  '
> + test -n Alejandro 'Aragon <address@hidden>'
> /bin/sh: line 0: test: Alejandro: binary operator expected

This indicates that your PACKAGE_BUGREPORT setting from configure.ac
contains weird kind of quoting, namely both double and single quotes.
Can you omit the single quotes?

Cheers,
Ralf