Re: valgrind-tests.m4

[Simon Josefsson]

Ralf Wildenhues <address@hidden> writes:

> Hi Simon,
>
> * Simon Josefsson wrote on Mon, May 17, 2010 at 12:15:51PM CEST:
>> I'm running self-tests under valgrind in several projects, and it seems
>> like a general useful thing, so how about putting the macro for this in
>> gnulib?
>
> Nice idea.  However, in general, tests might be executable programs, or
> they might be shell or other scripts.  In the latter cases, putting
> valgrind in TESTS_ENVIRONMENT would be at least a waste (you don't want
> to check bash or perl).

It is a waste, but I've found that it is a small price to pay for
valgrind test coverage.  Most of my projects doesn't mix too many shell
scripts with too many C files in the same test directory, too, so it has
been a marginal problem for me.

Of course, solving this concern would be nice, but it isn't clear to me
how to do it.  We can document the problem meanwhile and let maintainers
opt-in under that caveat.

> With Automake 1.11's parallel-tests option, you have the possibility
> to specify per-extension "compilers" for tests.
>
> Example:
>   # .sh and .pl files are processed to .log files.
>   TEST_EXTENSIONS = .sh .pl
>   SH_LOG_COMPILER = bash -vx
>   PL_LOG_COMPILER = perl -w -m strict
>   TESTS = foo.sh bar.pl baz.sh

I don't see how to use this for running valgrind only on compiled .c
files though.  Or are you suggesting inventing some "virtual" extension
like .valgrind?  I'm not sure how to use that with the intended
behaviour.

> Besides, TESTS_ENVIRONMENT really is meant as a variable for the
> end-user rather than for the developer.

That isn't clear from the Automake documentation.  I suspect it is will
be an uphill battle to try and fix that now.  The only example given for
TESTS_ENVIRONMENT in the example appears to be a maintainer-rule use:

   The variable `TESTS_ENVIRONMENT' can be used to set environment
   variables for the test run; the environment variable `srcdir' is set
   in the rule.  If all your test programs are scripts, you can also set
   `TESTS_ENVIRONMENT' to an invocation of the shell (e.g.  `$(SHELL)
   -x' can be useful for debugging the tests), or any other interpreter.
   For instance the following setup is used by the Automake package to
   run four tests in Perl.
     TESTS_ENVIRONMENT = $(PERL) -Mstrict -I $(top_srcdir)/lib -w
     TESTS = Condition.pl DisjConditions.pl Version.pl Wrap.pl

Gnulib makes use of TESTS_ENVIRONMENT in many places, as does Libtool
and, I suggest, several other GNU packages.

/Simon