[Bug-gnulib] Re: tracing from aclocal

[Alexandre Duret-Lutz]

[This mail contains patches for autoconf and automake interlaced
with comments.]

>>> "Akim" == Akim Demaille <address@hidden> writes:

[...]

 Akim> Any means that could guide people in writing proper M4
 Akim> should be used.

Ok, I'll work on this warning.

[...]

 Akim> Still, introducing an intermediate "language", without
 Akim> "aclocal.m4?" would do the same, but cleaner.

Here is a patch for this.  As always, a better name is welcome.

2003-08-18  Alexandre Duret-Lutz  <address@hidden>

        * lib/autom4te.in (Autoconf): Move all args except aclocal.m4? into ...
        (Autoconf-without-aclocal-m4): ... this new language.
        * doc/autoconf.texi (autom4te Invocation): Mention
        Autoconf-without-aclocal-m4.

Index: doc/autoconf.texi
===================================================================
RCS file: /cvsroot/autoconf/autoconf/doc/autoconf.texi,v
retrieving revision 1.752
diff -u -r1.752 autoconf.texi
--- doc/autoconf.texi   18 Aug 2003 16:48:26 -0000      1.752
+++ doc/autoconf.texi   18 Aug 2003 20:21:12 -0000
@@ -7914,6 +7914,10 @@
 
 @item Autoconf
 create Autoconf executable configure scripts.
+
address@hidden Autoconf-without-aclocal-m4
+create Autoconf executable configure scripts without
+reading @file{aclocal.m4}.
 @end table
 
 @item address@hidden
Index: lib/autom4te.in
===================================================================
RCS file: /cvsroot/autoconf/autoconf/lib/autom4te.in,v
retrieving revision 1.18
diff -u -r1.18 autom4te.in
--- lib/autom4te.in     25 Jun 2003 13:38:54 -0000      1.18
+++ lib/autom4te.in     18 Aug 2003 20:21:12 -0000
@@ -156,22 +156,33 @@
 end-language: "Autoscan-preselections"
 
 
-## ---------- ##
-## Autoconf.  ##
-## ---------- ##
+## ----------------------------- ##
+## Autoconf without aclocal.m4.  ##
+## ----------------------------- ##
 
-begin-language: "Autoconf"
+# This intermediate language is used by aclocal to build aclocal.m4.
+
+begin-language: "Autoconf-without-aclocal-m4"
 args: --prepend-include @datadir@
 args: --cache=autom4te.cache
 args: autoconf/autoconf.m4f
 args: acsite.m4?
-args: aclocal.m4?
 args: --mode 777
 args: --language Autoheader-preselections
 args: --language Automake-preselections
 args: --language Autoreconf-preselections
 args: --language Autoscan-preselections
 args: --language M4sh
+end-language: "Autoconf-without-aclocal-m4"
+
+
+## ---------- ##
+## Autoconf.  ##
+## ---------- ##
+
+begin-language: "Autoconf"
+args: --language Autoconf-without-aclocal-m4
+args: aclocal.m4?
 end-language: "Autoconf"
 
 

[...]

 >> As I understand it, this is because the second version of
 >> aclocal.m4 overrides the first within the same second, so
 >> autom4te sees the same timestamp.  If aclocal sleeps for
 >> 1sec between both writes, then caching does not occur (as
 >> expected).

 Akim> Indeed.  That's something Paul Eggert tracked down, and solved in
 Akim> autoreconf:

 Akim> if (!$uses_aclocal)
 Akim> {
 Akim> verbose "$configure_ac: not using aclocal";
 Akim> }
 Akim> else
 Akim> {
 Akim> # Some filesystems have sub-second time stamps, and if so we may
 Akim> # run into trouble later, after we rerun autoconf and set the
 Akim> # time stamps of input files to be no greater than aclocal.m4,
 Akim> # because the time-stamp-setting operation (utime) has a
 Akim> # resolution of only 1 second.  Work around the problem by
 Akim> # ensuring that there is at least a one-second window before the
 Akim> # time stamp of aclocal.m4t in which no file time stamps can
 Akim> # fall.
 Akim> sleep 1;

 Akim> run_aclocal ($aclocal, $aclocal_flags);
 Akim> }

If I follow this comment correctly, the sleep(1) is useless
with CVS aclocal.  Today it could as well be written

   sleep 1 unless $aclocal_supports_force;

couldn't it?  Because utime() is not used if $aclocal_supports_force.

That seems a different issue.  This one is caused by the use of
utime() which can _set_ the time stamp to an value older than its
original setting; while I'm somehow talking about the converse:
autom4te misses updates because it is _reading_ time stamps with
a 1sec resolution.

 Akim> Is this really all that can do a modern system?  A one
 Akim> second accuracy on time stamps?

Can't answer that one, but why focus on time stamps?  autom4te
could use checksums instead (at least when time stamps look
equal).

 >> In the case where we switch to your trick to compute the
 >> intermediate aclocal.m4, then there is no chance that this
 >> intermediate aclocal.m4 be equals to the final aclocal.m4.  The
 >> cache ought to be always invalided after aclocal has run.

 Akim> M4 is currently a serious bottleneck in our tool chain.  I'm not found
 Akim> of making this worse.  I might not understand fully what you are
 Akim> referring to (maybe it's time to have a diner together :), but it
 Akim> seems to me that that would introduce superfluous runs in some cases.

We have two runs of autom4te.  One during aclocal, one during autoconf.

In the first run, many useless files are read

   configure.ac m4/a.m4 m4/b.m4 /usr/share/aclocal/c.m4 /usr/share/aclocal/d.m4

In the second run, only a subset of these files remain, plus aclocal.m4

   configure.ac aclocal.m4 m4/a.m4
(aclocal.m4 might contain a copy of d.m4, and m4_include a.m4)

Essentially what I'm saying is that the second run cannot share
the cache of the first run, because we do not know about the
side effects of all the omitted macros.  (Considering the output
of --trace=AC_DEFUN in both cases might be another
justification.)


 >> Hence I think it would make sense to simply fix step 3. above to

 >> 3. autom4te --no-cache --language=autoconf --trace=... configure.ac 
 >> aclocal.t2
 >> ^^^^^^^^^^

 >> Sounds like (I still haven't tested) that would cure all issues,
 >> and at least that doesn't require any change to Autoconf.

 Akim> That should work.

Here is an updated patch.  It uses the new language introduced
above, and do not use any temporary file (such as the
aclocal.t2): aclocal just passes all the M4 files on the
autom4te command line, as in your example.

2003-08-18  Alexandre Duret-Lutz  <address@hidden>

        * aclocal.in (write_aclocal): Take an output file and a list of
        used macros in arguments and make up the file contents here.
        (trace_used_macros): New function.
        (add_file): Do not update $output.
        ($output): Delete.
        (MAIN): Call trace_used_macros.  Then rewrite aclocal.m4 only
        for these traced macros.  This should shorten aclocal.m4 by
        stripping out unused macros.
        * tests/aclibobj.test: Make sure configure.in exists by the time
        aclocal runs.
        * tests/aclocal8.test: New file.
        * tests/Makefile.am (TESTS): Add aclocal8.test.
        Suggested by Bruno Haible and Akim Demaille.

Index: NEWS
===================================================================
RCS file: /cvs/automake/automake/NEWS,v
retrieving revision 1.221
diff -u -r1.221 NEWS
--- NEWS        27 Jul 2003 12:58:38 -0000      1.221
+++ NEWS        18 Aug 2003 21:49:26 -0000
@@ -71,6 +71,11 @@
     Autom4te's cache isn't needlessly invalidated.  This behavior can
     be switched off with the new `--force' option.
 
+  - aclocal now uses Autoconf's --trace to detect macros which are actually
+    used and will no longer include unused macros simply because they
+    where mentioned.  This was often the case for macros called
+    conditionally.
+
   - New option no-dist-gzip.
 
   - install-sh now understands --version and --help.
Index: aclocal.in
===================================================================
RCS file: /cvs/automake/automake/aclocal.in,v
retrieving revision 1.84
diff -u -r1.84 aclocal.in
--- aclocal.in  7 Aug 2003 00:32:29 -0000       1.84
+++ aclocal.in  18 Aug 2003 21:49:27 -0000
@@ -55,9 +55,6 @@
 # Exit status.
 $exit_status = 0;
 
-# Text to output.
-$output = '';
-
 # Output file name.
 $output_file = 'aclocal.m4';
 
@@ -100,9 +97,10 @@
 &scan_m4_files (@dirlist);
 &scan_configure;
 if (! $exit_status)
-{
-    &write_aclocal;
-}
+  {
+    my %macro_traced = &trace_used_macros;
+    &write_aclocal ($output_file, keys %macro_traced);
+  }
 &check_acinclude;
 
 exit $exit_status;
@@ -375,22 +373,6 @@
     return if ($file_seen{$file});
     $file_seen{$file} = 1;
 
-    my $mtime = mtime $file;
-    $greatest_mtime = $mtime if $greatest_mtime < $mtime;
-
-    # If the file to add looks like path outside the project,
-    # copy it to the output.
-    # The regex catches filenames starting with things like
-    #   / \ c:\ ../ ./../ etc.
-    if ($file =~ m,^(?:(?:\w:)?[\\/]|(?:\.[\\/]+)*\.\.[\\/]),)
-      {
-       $output .= $file_contents{$file} . "\n";
-      }
-    else
-      {
-       # Otherwise, simply include the file.
-       $output .= "m4_include([$file])\n";
-      }
     my (@rlist);
     foreach (split ("\n", $file_contents{$file}))
     {
@@ -456,22 +438,75 @@
     return $contents;
 }
 
+sub trace_used_macros ($)
+{
+  my ($filename) = @_;
+
+  my %files = map { $map{$_} => 1 } keys %macro_seen;
+
+  my $traces = ($ENV{AUTOM4te} || 'autom4te');
+  $traces .= " --language Autoconf-without-aclocal-m4 --no-cache $filename ";
+  # All candidate files.
+  $traces .= join (' ', sort keys %files) . " ";
+  # All candidate macros.
+  $traces .= join (' ', map { "--trace='$_:\$n'" } (keys %macro_seen));
+
+  print STDERR "aclocal: running $traces $configure_ac\n" if $verbose;
+
+  my $tracefh = new Automake::XFile ("$traces $configure_ac |");
+
+  my %traced = ();
+
+  while ($_ = $tracefh->getline)
+    {
+      chomp;
+      $traced{$_} = 1 if $macro_seen{$_};
+    }
+  return %traced;
+}
+
 ################################################################
 
 # Write output.
-sub write_aclocal ()
+sub write_aclocal ($@)
 {
-    # Nothing to output?!
-    # FIXME: Shouldn't we diagnose this?
-    return if ! length ($output);
-
-# We used to print `# $output_file generated automatically etc.'  But
-# this creates spurious differences when using autoreconf.  Autoreconf
-# creates aclocal.m4t and then rename it to aclocal.m4, but the
-# rebuild rules generated by Automake create aclocal.m4 directly --
-# this would gives two ways to get the same file, with a different
-# name in the header.
-    $output = "# generated automatically by aclocal $VERSION -*- Autoconf -*-
+  my ($output_file, @macros) = @_;
+  my $output = '';
+
+  my %files = map { $map{$_} => 1 } @macros;
+  $files{'acinclude.m4'} = 1 if -f 'acinclude.m4';
+
+  for $file (sort keys %files)
+    {
+      my $mtime = mtime $file;
+      $greatest_mtime = $mtime if $greatest_mtime < $mtime;
+
+      # If the file to add looks like path outside the project,
+      # copy it to the output.
+      # The regex catches filenames starting with things like
+      #   / \ c:\ ../ ./../ etc.
+      if ($file =~ m,^(?:(?:\w:)?[\\/]|(?:\.[\\/]+)*\.\.[\\/]),)
+       {
+         $output .= $file_contents{$file} . "\n";
+       }
+      else
+       {
+         # Otherwise, simply include the file.
+         $output .= "m4_include([$file])\n";
+       }
+    }
+
+  # Nothing to output?!
+  # FIXME: Shouldn't we diagnose this?
+  return if ! length ($output);
+
+  # We used to print `# $output_file generated automatically etc.'  But
+  # this creates spurious differences when using autoreconf.  Autoreconf
+  # creates aclocal.m4t and then rename it to aclocal.m4, but the
+  # rebuild rules generated by Automake create aclocal.m4 directly --
+  # this would gives two ways to get the same file, with a different
+  # name in the header.
+  $output = "# generated automatically by aclocal $VERSION -*- Autoconf -*-
 
 # Copyright (C) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003
 # Free Software Foundation, Inc.
@@ -486,29 +521,30 @@
 
 $output";
 
-    # We try not to update $output_file unless necessary, because
-    # doing so invalidate Autom4te's cache and therefore slows down
-    # tools called after aclocal.
-    #
-    # We need to overwrite $output_file in the following situations.
-    #   * The --force option is in use.
-    #   * One of the dependencies is younger.
-    #     (Not updating $output_file in this situation would cause
-    #     make to call aclocal in loop.)
-    #   * The contents of the current file are different from what
-    #     we have computed.
-    if (!$force_output
-       && $greatest_mtime < mtime ($output_file)
-       && $output eq contents ($output_file))
-      {
-       print STDERR "aclocal: $output_file unchanged\n" if $verbose;
-       return;
-      }
-
-    print STDERR "aclocal: writing $output_file\n" if $verbose;
-
-    my $out = new Automake::XFile "> $output_file";
-    print $out $output;
+  # We try not to update $output_file unless necessary, because
+  # doing so invalidate Autom4te's cache and therefore slows down
+  # tools called after aclocal.
+  #
+  # We need to overwrite $output_file in the following situations.
+  #   * The --force option is in use.
+  #   * One of the dependencies is younger.
+  #     (Not updating $output_file in this situation would cause
+  #     make to call aclocal in loop.)
+  #   * The contents of the current file are different from what
+  #     we have computed.
+  if (!$force_output
+      && $greatest_mtime < mtime ($output_file)
+      && $output eq contents ($output_file))
+    {
+      print STDERR "aclocal: $output_file unchanged\n" if $verbose;
+      return;
+    }
+
+  print STDERR "aclocal: writing $output_file\n" if $verbose;
+
+  my $out = new Automake::XFile "> $output_file";
+  print $out $output;
+  return;
 }
 
 ### Setup "GNU" style for perl-mode and cperl-mode.
Index: tests/Makefile.am
===================================================================
RCS file: /cvs/automake/automake/tests/Makefile.am,v
retrieving revision 1.509
diff -u -r1.509 Makefile.am
--- tests/Makefile.am   9 Aug 2003 22:34:30 -0000       1.509
+++ tests/Makefile.am   18 Aug 2003 21:49:27 -0000
@@ -10,6 +10,7 @@
 aclocal5.test \
 aclocal6.test \
 aclocal7.test \
+aclocal8.test \
 acoutnoq.test \
 acoutpt.test \
 acoutpt2.test \
Index: tests/aclibobj.test
===================================================================
RCS file: /cvs/automake/automake/tests/aclibobj.test,v
retrieving revision 1.3
diff -u -r1.3 aclibobj.test
--- tests/aclibobj.test 8 Sep 2002 13:07:54 -0000       1.3
+++ tests/aclibobj.test 18 Aug 2003 21:49:27 -0000
@@ -1,5 +1,5 @@
 #! /bin/sh
-# Copyright (C) 2001, 2002  Free Software Foundation, Inc.
+# Copyright (C) 2001, 2002, 2003  Free Software Foundation, Inc.
 #
 # This file is part of GNU Automake.
 #
@@ -22,12 +22,10 @@
 
 . ./defs || exit 1
 
-cat > X << 'END'
-AC_INIT
-AM_INIT_AUTOMAKE(nonesuch, nonesuch)
+cat >> configure.in << 'END'
 AC_PROG_CC
 AC_PROG_RANLIB
-AC_OUTPUT(Makefile)
+AC_OUTPUT
 END
 
 cat > Makefile.am << 'END'
@@ -43,7 +41,7 @@
 
 set -e
 
-cp X configure.in
+cp configure.in X
 echo 'AC_LIBSOURCE(maude.c)' >> configure.in
 $AUTOMAKE
 
Index: tests/aclocal8.test
===================================================================
RCS file: tests/aclocal8.test
diff -N tests/aclocal8.test
--- /dev/null   1 Jan 1970 00:00:00 -0000
+++ tests/aclocal8.test 18 Aug 2003 21:49:27 -0000
@@ -0,0 +1,44 @@
+#! /bin/sh
+# Copyright (C) 2003  Free Software Foundation, Inc.
+#
+# This file is part of GNU Automake.
+#
+# GNU Automake is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# GNU Automake is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Automake; see the file COPYING.  If not, write to
+# the Free Software Foundation, Inc., 59 Temple Place - Suite 330,
+# Boston, MA 02111-1307, USA.
+
+# Make sure aclocal require unused macros.
+
+. ./defs || exit 1
+
+set -e
+
+cat >> configure.in << 'END'
+SOME_DEFS
+END
+
+mkdir m4
+cat >m4/somedefs.m4 <<EOF
+AC_DEFUN([SOME_DEFS], [
+  m4_if([a], [a], [MACRO1], [MACRO2])
+])
+EOF
+
+echo 'AC_DEFUN([MACRO1],)' >m4/macro1.m4
+echo 'AC_DEFUN([MACRO2],)' >m4/macro2.m4
+
+$ACLOCAL -I m4
+grep macro1.m4 aclocal.m4
+grep macro2.m4 aclocal.m4 && exit 1
+:

[...]

-- 
Alexandre Duret-Lutz