Re: calling autoreconf and minimizing rebuilds

[Ralf Wildenhues]

Hello Stepan,

* Stepan Kasal wrote on Fri, Sep 08, 2006 at 04:11:22PM CEST:
> On Fri, Sep 08, 2006 at 07:16:06AM +0200, Ralf Wildenhues wrote:
> > [...], with Stepan's proposed additional patch
> > http://lists.gnu.org/archive/html/autoconf-patches/2006-09/msg00023.html
> > I can't get the test to fail reliably any more [...]

> my guess is that it all happens too fast;

Yes.  But that's not the complete answer.

> If all of the above takes place within the same second, and if the
> file system does not support sub-second timestamps, then configure
> has the same timestamp as output.1 and traces.1.

Indeed my GNU/Linux system stores only 1-second timestamps on disk.

Adding some sleep in between can fix this.  But also I kept seeing false
negatives from time to time.  Let's see: autom4te has this:

|    # Actual M4 expansion, only if $output is too old. STDOUT is
|    # pretty old.
|    handle_output ($req, $output)
|      if mtime ($output) < mtime ($ocache . $req->id);

But aclocal works differently: it updates it output when any of its
inputs are not older:

|   # We need to overwrite $output_file in the following situations.
|   #   * The --force option is in use.
|   #   * One of the dependencies is younger.
|   #     (Not updating $output_file in this situation would cause
|   #     make to call aclocal in loop.)
[...]
|   if (!$force_output
|       && $greatest_mtime < mtime ($output_file)
|       && $output eq contents ($output_file))
|     {
|       verb "$output_file unchanged";
|       return 1;
|     }


> Actually, I did not imagine that the problem will be reproduced by a
> call to autoreconf, I imagined the individual tools would be called.
> (When I was analysing the problem, I did not use autoreconf either.)

Well, yes, but: the original problem was reported with autoreconf, so
why should we not model it after that?  IOW: we want that to work which
the user uses.

> (And yes, it is quite possible that the patch quoted above makes this
> procedure faster, increasing  the probability that it all falls to
> the same second.)

So the increased failure rate with your patch would be a sign that your
Autoconf patch is indeed an optimization ... well.  It suggests that
your patch increased at least one part of the autotools, but makes no
statement about its weighted sum (weighted by average usage count).

> More comments:
> > +# If [...] the user has a local config, skip:
> ...
> > +test -f $HOME/.autom4te.cfg && exit 77
> 
> I do not understand why this is necessary, but it is not a big deal.

Well, the user can turn off autom4te caching by writing to this file.
Or he could overwrite the language preselections here, right?
I doubt that anyone would do the latter (and I intended to catch the
former further below), but better be safer than sorry: this test does
not need to be executed on many systems in order to be effective.

> For the tail of the script, I would use something like:
> 
> $ACLOCAL
> $AUTOCONF
> : >stamp
> $sleep
> $AUTOMAKE
> test -z "`find configure -newer stamp`"

Running automake never causes an updated `configure' script.
That would be a severe bug (it would confuse `make').

Also, Automake tests use `ls -1t' a lot, but never `find -newer',
and I assume there is a reason for that; in any case I prefer to be
consistent.

> > +if $AUTORECONF; then :; else
> > +  if test $? -eq 63 || test ! -d autom4te.cache; then
> > +    exit 77
> > +  fi
> > +fi
> 
> What if autoconf and autoreconf succeed but caching is switched off?
> The test for -d autom4te.cache is never reached and the script will
> fail later.

Thanks for catching this bug.

> I would suggest this (based on my above code):

> $ACLOCAL
> if $AUTOCONF; then
>   test -d autom4te.cache || exit 77
> else
>   status=$?
>   test $status != 63 || status=77
>   exit $status
> fi
> : >stamp
> $sleep
> $AUTOMAKE
> test -z "`find configure -newer stamp`"

Closer.  We need to sleep before the first aclocal, and sleep before the
stamp (otherwise it's not guaranteed to be younger than configure), and
$status is forbidden.  And aclocal already fails with 63 if autoconf is
too old.  And I would like to be stronger (compare aclocal7.test) in
that aclocal should not cause autoconf to rebuild configure either.
Finally, using && and || inside if-then or else clauses is unhealthy
with `set -e' on some BSD shells.

With the patch below, I cannot produce false positives nor false
negatives any more.  OK (as two separate patches)?

Cheers,
Ralf

        * configure.ac: Preserve original values of `$AUTORECONF'
        and `$AUTOHEADER', to avoid the `missing' script.
        * tests/defs.in: Adjust.

        * tests/trace.test: New test for autom4te preselections.
        * tests/Makefile.am: Update.
        Report against Autoconf by David Byron <address@hidden>.

Index: configure.ac
===================================================================
RCS file: /cvs/automake/automake/configure.ac,v
retrieving revision 1.32
diff -u -r1.32 configure.ac
--- configure.ac        20 Aug 2006 16:58:19 -0000      1.32
+++ configure.ac        8 Sep 2006 19:26:18 -0000
@@ -30,7 +30,9 @@
 # Save the AUTOCONF setting before AM_INIT_AUTOMAKE overrides it; this
 # way we can run Autoconf tests from configure (or from the test
 # suite) without being bothered by `missing'.
+# Likewise for AUTOHEADER.
 AC_SUBST([am_AUTOCONF], ["${AUTOCONF-autoconf}"])
+AC_SUBST([am_AUTOHEADER], ["${AUTOHEADER-autoheader}"])
 
 AM_INIT_AUTOMAKE([1.8a dist-bzip2 filename-length-max=99])
 
Index: tests/defs.in
===================================================================
RCS file: /cvs/automake/automake/tests/defs.in,v
retrieving revision 1.39
diff -u -r1.39 defs.in
--- tests/defs.in       6 Jul 2006 18:13:01 -0000       1.39
+++ tests/defs.in       8 Sep 2006 19:26:25 -0000
@@ -73,7 +73,7 @@
 test -z "$PERL" && PERL='@PERL@'
 test -z "$MAKE" && MAKE=make
 test -z "$AUTOCONF" && AUTOCONF="@am_AUTOCONF@"
-test -z "$AUTOHEADER" && AUTOHEADER="@AUTOHEADER@"
+test -z "$AUTOHEADER" && AUTOHEADER="@am_AUTOHEADER@"
 test -z "$AUTOUPDATE" && AUTOUPDATE=autoupdate
 test -z "$MISSING" && MISSING=`pwd`/../lib/missing
 # Use -Werror because this also turns some Perl warnings into error.
Index: tests/Makefile.am
===================================================================
RCS file: /cvs/automake/automake/tests/Makefile.am,v
retrieving revision 1.609
diff -u -r1.609 Makefile.am
--- tests/Makefile.am   30 Aug 2006 20:35:56 -0000      1.609
+++ tests/Makefile.am   8 Sep 2006 19:26:25 -0000
@@ -542,6 +543,7 @@
 tar3.test \
 target-cflags.test \
 targetclash.test \
+trace.test \
 txinfo.test \
 txinfo2.test \
 txinfo3.test \
--- /dev/null   2006-09-05 22:40:33.520458500 +0200
+++ tests/trace.test    2006-09-08 21:10:25.000000000 +0200
@@ -0,0 +1,65 @@
+#! /bin/sh
+# Copyright (C) 2006  Free Software Foundation, Inc.
+#
+# This file is part of GNU Automake.
+#
+# GNU Automake is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# GNU Automake is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Automake; see the file COPYING.  If not, write to
+# the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor,
+# Boston, MA 02110-1301, USA.
+
+# Ensure that the macros traced by automake are listed in autom4te.cfg.
+# If all of them are listed, and autom4te caching works as intended,
+# then configure should not be regenerated.
+
+. ./defs || exit 1
+
+set -e
+
+# If `--force' is used, or the user has a local config, skip:
+for arg in $AUTOCONF
+do
+  case $arg in
+  --force | -f* | -[!-]*f*) exit 77;;
+  esac
+done
+test -f $HOME/.autom4te.cfg && exit 77
+
+cat >> configure.in <<EOF
+AC_PREREQ(2.60b) dnl Fixed autom4te.cfg should be in this version.
+EOF
+cat >Makefile.am <<EOF
+AUTOMAKE_OPTIONS = foreign
+EOF
+
+$sleep # `aclocal.m4' should be strictly younger than its inputs
+
+# If Autoconf is too old, or the user has turned caching off, skip:
+if $ACLOCAL; then :; else
+  ret=$?
+  if test $ret -eq 63; then
+    ret=77
+  fi
+  exit $ret
+fi
+test -d autom4te.cache || exit 77
+$AUTOCONF
+$sleep # need to make sure the stamp file is strictly newer.
+echo newer >newer
+$sleep # if `configure' is regenerated, we want it to be strictly newer,
+       # to catch the error consistently.
+$ACLOCAL
+$AUTOMAKE --no-force
+$AUTOCONF
+set x `ls -1t newer configure`
+test "$2" = newer