automake
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: running tests under in a tests/ directory


From: aaragon
Subject: Re: running tests under in a tests/ directory
Date: Sat, 30 May 2009 10:34:13 -0700 (PDT)

Thank yo Ralf for replying to my post. The code that I showed in my post was
indeed taken from that website. The code you provided me is way too advanced
for me, since I consider myself just a user. The main idea that I got from
your reply is that this new code won't work with older versions of Automake.
Is this the case? Can something be done regarding this issue? Let say, check
for the version of Automake and if it is indeed version 1.11 then use the
newer features but if the version is older, then fall back to something that
can still be used. Otherwise the code that you wrote is not really portable.
For example, I compile my code in a cluster that has version 1.9.2 installed
and I have no way to upgrade that.

Now, regarding the code that you wrote, I am trying to make it work, so I
changed the Makefile.am in the root directory to what you suggested
(almost):

# test rule
TSTDIR = tests

check:  all
        @echo 
"-------------------------------------------------------------------"
        @echo "Running tests"
        cd $(TSTDIR) && $(MAKE) $(AM_MAKEFLAGS) -s test
        @echo 
"-------------------------------------------------------------------"

Except that if I change 'check: all' to something else, then it doesn't work
anymore.

In the tests directory, I changed the Makefile.am to reflect that I use only
cpp files. I also made the generate-verified-files silent, so that the
output of that target is:

address@hidden/Documents/workspace/cpputils/tests$make generate-verified-files
 cd .. && /bin/sh /Users/aaragon/Documents/workspace/cpputils/config/missing
--run automake-1.11 --gnu tests/Makefile
 cd .. && /bin/sh ./config.status tests/Makefile depfiles
config.status: creating tests/Makefile
config.status: executing depfiles commands
make -s  AM_LOG_FLAGS=--create check
PASS: test001
PASS: testcppblas
==================
All 2 tests passed
==================

Now, from the code that I got from the internet, I really liked the fact
that you don't need to specify the name of the tests in the Makefile.am file
because of the use of wildcards. Is there a way to make something similar
here that is really portable? In the end, all tests will have the .cpp
extension in the directory, so there has to be a way to determine all files
without specifying them on Makefile.am explicitly.

If I make one of the tests fail (by deliberately changing one of the
verified files), I get a message saying that a binary operator is expected:

address@hidden/Documents/workspace/cpputils/tests$make check
make  check-TESTS
FAIL: test001
PASS: testcppblas
/bin/sh: line 0: test: Alejandro: binary operator expected
========================
1 of 2 tests failed
See tests/test-suite.log
========================
make[2]: *** [test-suite.log] Error 1
make[1]: *** [check-TESTS] Error 2
make: *** [check-am] Error 2

Is this normal?

Then I make the target to generate verified files (which I may change to
'make verified' in the future because it's too long)

address@hidden/Documents/workspace/cpputils/tests$make generate-verified-files
make -s  AM_LOG_FLAGS=--create check
PASS: test001
PASS: testcppblas
==================
All 2 tests passed
==================
address@hidden/Documents/workspace/cpputils/tests$make check
make  check-TESTS
PASS: test001
PASS: testcppblas
==================
All 2 tests passed
==================

I really like how the code that you wrote works. Thanks a lot and I hope you
reply to this post soon.

Alejandro


Ralf Wildenhues wrote:
> 
> [ <http://thread.gmane.org/gmane.comp.sysutils.automake.general/10749> ]
> 
> Hello Alejandro, Martin,
> 
> * aaragon wrote on Fri, May 29, 2009 at 05:44:57AM CEST:
>> 
>> I found a Makefile.am file on the internet to run tests under a tests
>> folder. The code looks like follows:
> 
> [ *snip*, looks an awful lot like
>   <http://www.bioinf.uni-freiburg.de/~mmann/HowTo/automake.html> ]
> 
> Martin, a while ago I went over your Automake page and noted some issues
> with it, and proposed a newer version that uses Automake 1.11 features.
> 1.11 has been released now, would you be so nice to update your web page
> to reflect these fixes?
> 
> A few nits inline below, and part of my recommendation follows at the
> end.  It will also help you, Alejandro, avoid the pitfall of letting
> code be compiled by GNU make implicit rules (which uses different
> variable names and thus does not pick up the include paths).
> 
>> check:       all
> 
> You can use check-local; and check will depend upon all-am anyway (the
> part of 'all' that is updated in this directory), maybe that is
> sufficient for your needs.
> 
>>      @echo "Running tests"
>>      @make test -s -C $(TSTDIR)
> 
> Non-GNU make doesn't support -C, and you should use $(MAKE) so that
> parallel make and make -k and make -n work; you can use
>   cd $(TSTDIR) && $(MAKE) $(AM_MAKEFLAGS) -s test
> 
> but if I were you, I'd just let the automake-provided check rule do its
> work in the toplevel directory.
> 
>> TESTSOURCE = $(wildcard test*.cpp)
> 
> GNU make-specific.
> 
>> TESTOBJS=$(TESTSOURCE:%.cpp=%.o)
> 
> Likewise.  More instances below.
> 
>> # additional linker and compiler flags
>> LDFLAGS = -L$(top_srcdir)/yafeq -lyafeq $(LDFLAGS)
> 
> For in-tree libtool libraries, please always specify them as relative
> paths to the .la file; also, don't stick them in LDFLAGS; thus, use
>   LDADD = $(top_srcdir)/yafeq/libyafeq.la
> 
> (or even LIBS).
> 
>> address@hidden/Documents/workspace/yafeq/tests$mt
>>  Compiling IntegrationRules.cpp                   OK
>> BD Software STL Message Decryptor v3.10 for gcc 2/3/4
>> In file included from testIntegrationRules.cpp:1:
>> ../yafeq/quadrature.hpp:14:36: error: cpputils/map_adaptor.hpp: No such
>> file
>> or
>>     directory
>> 
>> 
>> Now, the file cpputils/map_adaptor.hpp does exist, and I actually have a
>> test for detecting the cpputils library. I guess the CPPFLAGS that
>> includes
>> the correct path is not being passed to this piece of code written in the
>> Makefile.am.
> 
> 
> What follows is my suggestion that I wrote to Martin about the tests
> thing.  I haven't tested it much, so if you have issues with them,
> please speak up.
> 
> Cheers,
> Ralf
> 
> For the tests example, I actually have a more radical suggestion:
> replace it completely with a setup based on automake support for simple
> tests using the TESTS variable.
> 
> Since I am lazy, and we have added several new features to the upcoming
> Automake 1.11 release which can be of help here, I will further suggest
> to require that next version, and use the 'parallel-tests' test driver
> which has several new features.  In the following, I will also use some
> more Automake 1.11 features like AM_DEFAULT_SOURCE_EXT.
> (The stuff can more or less also be done with 1.10 features, but it will
> be uglier, and, as already noted, I am lazy.  :-)
> 
> However, for portability, you will not be able to use wildcards.  So you
> will need to mention each test once.
> 
> BTW, in your example is a line of the form
> | LDFLAGS = $(LDFLAGS)
> 
> If this really were in your code, then it would provoke an endless
> recursion, and an error from 'make' if referenced anywhere.
> 
> | # additional linker and compiler flags
> | CXXFLAGS = -I$(top_srcdir)/src $(CXXFLAGS)
> 
> You should not override CXXFLAGS: that variable is reserved for the
> user.  Automake provides AM_CXXFLAGS for that case.  However, include
> paths are for the preprocessor, so AM_CPPFLAGS should be used here.
> 
> With the tests/Makefile.am and the test driver files below, there is no
> need to modify toplevel Makefile.am, and you can run the testsuite with
>   make check
> 
> or even
>   make -j3 check
> 
> if you so prefer.  :-)
> 
> A word of caution: this code is completely untested.
> 
> Another word of caution: you need to run
>   make generate-verified-files
> 
> once before you can run "make dist".  If you do not like that kind of
> setup, please speak up.  (The dependencies are not completely precise
> yet.)
> 
> You can update individual .verified files with something like
>   make generate-verified-files TESTS='test1 test2'
> 
> Please note that 'diff -w' is not portable, at least not POSIX.
> I simply used diff in the driver, but you may want to make that
> configurable or so.
> 
> Cheers,
> Ralf
> 
> --- tests/diff-driver ---
> #! /bin/sh
> 
> # Compare output of test programs with expected output.
> # Usage: diff-driver [--create] TEST-PROGRAM
> #
> # TEST-PROGRAM is the program to run.
> # With --create, a .verified file is created.
> # Without it, a .lastout file is created and compared
> # to an existing .verified file; the diff is produced on stdout.
> # $srcdir should contain the path to the source directory.
> #
> # .lastout files are created in the build tree; the TEST-PROGRAM
> # may or may not live in the source tree; an existing .verified
> # file may live in the source tree (if it came with the distribution)
> # or the build tree, but is updated in the build tree.
> 
> if test -z "$srcdir"; then
>   echo "$0: \`\$srcdir' is not set" >&2
>   exit 1
> fi
> 
> create=no
> for arg
> do
>   case $arg in
>     --create)
>       create=yes ;;
>     [!-]*)
>       break ;;
>   esac
>   shift
> done
> 
> test_program=$1
> test_program_noext=${test_program%$EXEEXT}
> case $test_program_noext in
>   $srcdir/*)
>     srcdirstrip=`echo "$srcdir" | sed 's|.|.|g'`
>     test_stem=`echo "$test_program_noext" | sed "s|^$srcdirstrip/||"` ;;
>   *)
>     test_stem=$test_program_noext ;;
> esac
> 
> output_file=$test_stem.lastout
> verified_file=$test_stem.verified
> if test "$create" = no && test ! -f "$verified_file"; then
>   verified_file=$srcdir/$verified_file
>   if test ! -f "$verified_file"; then
>     echo "$0: cannot find .verified file \`$verified_file'" >&2
>     exit 1
>   fi
> fi
> 
> if test "$create" = yes; then
>   $test_program > $verified_file
>   exit $?
> else
>   $test_program > $output_file || exit $?
>   diff $verified_file $output_file
>   exit $?
> fi
> 
> --- tests/Makefile.am ---
> 
> # We use the new parallel-tests driver from Automake 1.11
> # For nice and pretty output, we will be using color, too.
> AUTOMAKE_OPTIONS = 1.10c parallel-tests color-tests
> 
> # Each program `foo' is compiled from `foo.cc', not `foo.c'.
> AM_DEFAULT_SOURCE_EXT = .cc
> 
> # Add test programs here.
> # If their sources do not end in .cc, or are made up of more than one
> # file, then you need to provide a testX_SOURCES variable, too.
> TESTEXECS = test1 test2 test3 ...
> 
> # All test executables are to be run by `make check'.
> TESTS = $(TESTEXECS)
> 
> # Let automake be aware that all test executables are programs to be
> # compiled (and $(EXEEXT) appended etc) using the normal PROGRAMS
> # machinery.  We use EXTRA_ instead of check_, because the
> # parallel-tests driver will cause the programs to be built as normal
> # prerequisites just in time.
> EXTRA_PROGRAMS = $(TESTEXECS)
> 
> AM_CPPFLAGS = -I$(top_srcdir)/src
> 
> # Tests are run with the diff-driver
> LOG_COMPILER = ./diff-driver
> 
> # Run `make generate-verified-files' to create .verified files.
> # This just runs all tests and passes --create to the diff-driver.
> generate-verified-files:
>       $(MAKE) $(AM_MAKEFLAGS) AM_LOG_FLAGS=--create check
> 
> # add verified output files to the distribution
> EXTRA_DIST = $(TESTEXECS:=.verified)
> 
> MOSTLYCLEANFILES = $(TESTEXECS:=.lastout)
> MAINTAINERCLEANFILES = $(TESTEXECS:=.verified)
> 
> 
> 
> 
> 

-- 
View this message in context: 
http://www.nabble.com/running-tests-under-in-a-tests--directory-tp23773808p23795557.html
Sent from the Gnu - Automake - General mailing list archive at Nabble.com.





reply via email to

[Prev in Thread] Current Thread [Next in Thread]