patch series: fast install and uninstall

[Ralf Wildenhues]

Hello,

Continuing the patch series against git master Automake, to fix and
improve 'make install' and 'make uninstall' (and 'make clean' a bit).
There's about a dozen patches coming up.

Jim and Akim pushed towards multi-file 'install' because the GNU
coreutils install program is rather heavy, esp. with ACLs in the mix.
Thanks for the good hints.


A few observations:

* `make install' wasn't reliable so far, in that it could exit
successfully although installation really failed.  This is fixed
first (see [1] for a note about uninstall).

* Forks&execs are the most expensive bits of most make rule snippets,
and maybe quadratic scaling with long lists.

* In many cases, several programs are called once per installed file
(install, sed, subshells).  While 'install' is arguably the most
expensive, it's good to avoid the others, too.  So this series aims
to use a constant amount of forks per install rule only.

* Quadratic scaling is avoided completely when recent bash is used,
otherwise it is limited in practice by the following observation.

* It is important not to exceed command line length limits, which can
be quite low for some systems.  The patches that went in so far didn't
take this into account.  A number of measures are now taken:

- a new chapter in the manual about what Automake does to help, and what
  the user has to do and can do.  Some of this is old and long-known,
  some is new due to these patches.
- limiting the number of files installed at once, *whenever* the file
  names are rewritten (e.g., to contain '$(srcdir)/').  (If they are not
  rewritten, it does not make sense to limit, because then the original
  command line spawned by 'make' would have exceeded the limit already
  anyway.)
- a choice not to adapt the limiting to the system, and not to do
  precise length counting.  This is for simplicity, and it is expected
  that the slowdown due to this is rather small on modern systems[2].

Note that in practice, the rules require that record limits for sed and
awk have to be respected as well.

Between portability, speed, brevity, and code clarity, when a choice
was needed, I sacrificed the last (two) in order to keep the others.
Luckily, the amount of parallelism available is completely orthogonal
to these changes, so not looked at any further.

The stdout output may be a bit harder to read, but should otherwise
still be correct.  awk is used quite a bit more now, for its hash and
counting flexibility, similar to how Autoconf uses it more now.

For small packages, when only one or two files per rule are installed,
a small but really minor slowdown may be observed.  I think the turnover
really is at two files per rule already.

Here's some example timings, comparing Automake 1.10.1 with git Automake
plus the patch series (time in seconds, best of three, GNU/Linux system,
'make -s DESTDIR=...' install and uninstall).

           OpenMPI     Valgrind    Texinfo     Autoconf    Automake
           pre   post  pre   post  pre   post  pre   post  pre   post
install    52.7  41.4  4.81  3.93  2.21  1.84  1.34  0.78  2.25  0.65
uninstall  31.8  19.5  2.41  1.63  1.43  1.03  1.19  0.54  1.88  0.53
clean      21.8  21.8  1.87  1.81  0.50  0.54  0.36  0.33  0.18  0.15

OpenMPI isn't exactly a poster child for this because it uses
libtool a lot, and its build tree traversal is quite expensive.
Texinfo could benefit from the same optimization done in po/.
I didn't include gettext in the comparison because it has hand-
written rules for most of its installation.

The speedup for Automake itself is nice.  :-)

I hope to release an 1.10a alpha release not so long after finishing
the series.  Of course, then, but even before, I appreciate any
kind of test feedback, review, incompatibilities you experience
when using the new Automake on your packages.  The patches all
operate under the condition that user code is not impacted at all.
Now, that's difficult to achieve in practice, and NEWS will list
a couple of known small issues, but if you experience any glitch
then we'd like to know about it, so we can fix it before 1.11.

Thanks,
Ralf

[1] uninstall is somewhat of a different (and less problematic) beast.
However,
  make uninstall && make uninstall

should not fail.  This needs some further fixing first.

[2] Linux as of 2.6.23 has no limit any more, Hurd never had it.

PS: 'make dist' can probably be improved, too, but this is a less
pressing (and a bit more complicated) issue.

PPS: Libtool should learn multi-file install, too.  So far all `libtool
--mode=install' stuff is single-file.