parallel-tests: warn about $(srcdir), $(top_srcdir) in TESTS.

[Ralf Wildenhues]

* Ralf Wildenhues wrote on Fri, Mar 20, 2009 at 12:24:15AM CET:
>   set x foo*.test ../../source/dir/bar*.test
>   shift
>   make check "TESTS=$*"
> 
> or
>   eval make check \'TESTS= foo*.test ../../source/dir/bar*.test \'

Hrmpf.  Using $(srcdir) in TESTS is really bad: it causes the test log
to end up in the source tree.

For the above kind of hacks, there are workarounds, like
  cd $srcdir; set x foo*.test; shift; cd -
  make check TESTS="$*"

We could also rewrite TESTS to eliminate the $(srcdir) prefixes, which
would work for TESTS values passed to 'make', but:

The Makefile.am author might have a good reason to use explicit
$(srcdir) for generated distributed test files, quoting
(autoconf.info)Make Target Lookup:

|    OpenBSD and FreeBSD `make', however, never perform a `VPATH' search
| for a dependency that has an explicit rule.  This is extremely annoying.

In this case, rewriting TESTS will silently prevent `make' from
considering the rule to update the test.  So, let's not do that,
creating silent bugs, but instead error out if we see this.

I'm applying this patch to the ad-parallel-tests branch, also noting
the experimental status of this test driver, we might encounter more
such issues...

BTW, Solaris make VPATH rewriting is not a problem for this particular
issue: it only occurs if TESTS is actually listed as prerequisite to the
rule we're currently executing, but not in TEST_LOGS which is something
like $(TESTS:.test=.log), only more complicated.

Cheers,
Ralf

    parallel-tests: warn about $(srcdir), $(top_srcdir) in TESTS.
    
    * automake.in (handle_tests): Warn about portability issue
    concerning generated TESTS files listed with a `$(srcdir)/'
    or `$(top_srcdir)/' prefix.
    * doc/automake.texi (TESTS): Document this issue.  Mention that
    the parallel-tests driver is still experimental.
    * tests/parallel-tests8.test: New test.
    * tests/Makefile.am: Update.

diff --git a/automake.in b/automake.in
index 433f88e..ce20cce 100755
--- a/automake.in
+++ b/automake.in
@@ -4789,6 +4789,13 @@ sub handle_tests
                return $obj
                  if $val =~ /address@hidden@$/;
                $obj =~ s/\$\(EXEEXT\)$//o;
+
+               if ($val =~ /(\$\((top_)?srcdir\))\//o)
+                 {
+                   msg ('error', $subvar->rdef ($cond)->location,
+                        "parallel-tests: using `$1' in TESTS is currently 
broken: `$val'");
+                 }
+
                foreach my $test_suffix (@test_suffixes)
                  {
                    next
diff --git a/doc/automake.texi b/doc/automake.texi
index d3b72a5..bf41acb 100644
--- a/doc/automake.texi
+++ b/doc/automake.texi
@@ -8371,6 +8371,7 @@ by the tests, not the tests themselves.  Of course you 
can set
 
 @section Simple tests using @samp{parallel-tests}
 @cindex @option{parallel-tests}, Using
+
 The option @option{parallel-tests} (@pxref{Options}) enables a test
 suite driver that is mostly compatible to the simple test driver
 described above, but provides a few more features and slightly different
@@ -8383,6 +8384,9 @@ for exceptional failures.  Similar to the simple test 
driver,
 the @code{check_*} variables are honored, and the environment variable
 @env{srcdir} is set during test execution.
 
+This test driver is still experimental and may undergo changes in order
+to satisfy additional portability requirements.
+
 @vindex TEST_SUITE_LOG
 @vindex TEST_LOGS
 The driver operates by defining a set of @command{make} rules to create
@@ -8504,6 +8508,17 @@ the inference rules generated by @command{automake} can 
apply.  For
 literal test names, @command{automake} can generate per-target rules
 to avoid this limitation.
 
+Please note that it is currently not possible to use @code{$(srcdir)/}
+or @code{$(top_srcdir)/} in the @code{TESTS} variable.  This technical
+limitation is necessary to avoid generating test logs in the source tree
+and has the unfortunate consequence thast it is not possible to specify
+distributed tests that are themselves generated by means of explicit
+rules, in a way that is portable to all @command{make} implementations
+(@pxref{Make Target Lookup,,, autoconf, The Autoconf Manual}, the
+semantics of FreeBSD and OpenBSD @command{make} conflict with this).
+In case of doubt you may want to require to use GNU @command{make},
+or work around the issue with inference rules to generate the tests.
+
 
 @section DejaGnu Tests
 
diff --git a/tests/Makefile.am b/tests/Makefile.am
index 3f31ca9..daecf34 100644
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -486,6 +486,7 @@ parallel-tests4.test \
 parallel-tests5.test \
 parallel-tests6.test \
 parallel-tests7.test \
+parallel-tests8.test \
 parse.test \
 percent.test \
 percent2.test \
diff --git a/tests/parallel-tests8.test b/tests/parallel-tests8.test
new file mode 100755
index 0000000..309a2cf
--- /dev/null
+++ b/tests/parallel-tests8.test
@@ -0,0 +1,75 @@
+#! /bin/sh
+# Copyright (C) 2009  Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+# Check parallel-tests features:
+# - generated distributed tests.
+# - listing $(srcdir)/ or $(top_srcdir)/ in TESTS doesn't work ATM,
+#   and is thus diagnosed.
+
+# TODO: this test should also ensure that the `make' implementation
+#       properly adheres to rules in all cases.  See the Autoconf
+#       manual for the ugliness in this area, when VPATH comes into
+#       play.  :-/
+
+. ./defs-p || Exit 1
+
+set -e
+
+cat >> configure.in << 'END'
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TESTS = foo.test
+## the next line will cause automake to error out:
+TESTS += $(srcdir)/bar.test $(top_srcdir)/baz.test
+.in.test:
+       cp $< $@
+       chmod +x $@
+check_SCRIPTS = $(TESTS)
+EXTRA_DIST = foo.in foo.test
+DISTCLEANFILES = foo.test
+END
+
+cat >>foo.in <<'END'
+#! /bin/sh
+echo "this is $0"
+exit 0
+END
+
+$ACLOCAL
+$AUTOCONF
+AUTOMAKE_fails -a
+grep '(srcdir.*bar' stderr
+grep 'top_srcdir.*baz' stderr
+
+sed '/srcdir/d' < Makefile.am > t
+mv -f t Makefile.am
+$AUTOMAKE -a
+
+./configure
+$MAKE check
+$MAKE distcheck
+$MAKE distclean
+
+mkdir build
+cd build
+../configure
+$MAKE check
+test ! -f ../foo.log
+$MAKE distcheck
+
+: