test with old autotools (was: [PATCH 1/6] Require Automake 1.11.1 for Libtool, enable color-tests.)

[Ralf Wildenhues]

Hello Gary,

you wanted a way to test old Autoconf and Automake with a git Libtool
tree:

* Gary V. Vaughan wrote on Sun, Aug 22, 2010 at 05:41:52PM CEST:
> On 22 Aug 2010, at 22:30, Ralf Wildenhues wrote:
> > * Gary V. Vaughan wrote on Sun, Aug 22, 2010 at 05:24:05PM CEST:
> >> Of course, that doesn't test the patch series you haven't
> >> pushed yet...
> > 
> > Just add 'git checkout origin/parallel-tests'.
> 
> I meant that if I want to check that my next round of m4sh patches
> don't break with older Autotools before I push, this script doesn't
> help, since it wants a fresh libtool checkout.  The script would
> be more useful if it can be run inside a dirty working directory.
> 
> >> And running it as an Autotest inside the existing tree would
> >> get us better coverage, and obviate the need for a separate
> >> checkout of the libtool tree entirely.
> > 
> > Sure.  But look at the existing sort-of-recursive tests.  They are
> > tricky.  I am waiting for the day where we add infinite recursion in
> > some corner case and have developers complain.
> 
> Actually, I've not looked at the details of those recursive tests,
> so I must admit that I don't know how tricky they are.  I just
> assumed that we could reuse that pattern here.
> 
> > The net is abundant now, I think such test helper scripts are a real
> > easy way to get more coverage, but I don't think it is necessary to
> > run them as part of each testsuite run, because they are very expensive.
> 
> A very good point.  Maybe we should add a 'maintainer-test' target or
> similar, which executes this and other expensive test-helper scripts
> in addition to the regular testsuite(s). And it's also worth considering
> migrating our other recursive tests to helper scripts too in that case.

I haven't done that yet; but in the end, that might actually be a better
idea than what I hacked up for now:

This patch checks whether $AUTOCONF and $AUTOMAKE point to the oldest
versions we aim to support, otherwise downloads repective release
tarballs, builds and installs them temporarily below a subdirectory in
the test group, then runs those parts of the new testsuite recursively
which use autoconf.  The old testsuite is not dealt with.

While debugging the patch, I did run into an endless recursion or two.
The test is fragile in that it might need updating when we add more
environment variables that influence general test behavior.

At this point, I'm not actually asking to commit this now.  Consider it
a test balloon that you could use in your git tree to test things.  :-)

Cheers,
Ralf

0001-Try-out-new-testsuite-with-old-Autoconf-and-Automake.patch
Description: Text document