other/42980: fix make -j install

[Ralf Wildenhues]

'make -jN install' currently has a few race conditions causing
intermittent failures.

- <http://gcc.gnu.org/bugzilla/show_bug.cgi?id=33119>
  which has been fixed since, AFAICS, but
- <http://gcc.gnu.org/bugzilla/show_bug.cgi?id=42980>
  exposes a couple of separate issues:

a) The toplevel targets install-libiberty and install-target-libiberty
may both be executed, and in that case, the install_to_libdir rules from
libiberty/ may race with the same rules from $target/libiberty/ (quite
likely) or with those from a $target$MULTISUBDIR/libiberty (a lot less
likely, only a problem for header installation).

b) libgcc has a make recursion lacking $(MAKE), needed for parallel
submake.

c) Inside libgomp, libssp and libmudflap (and possibly other target
libraries), data (non-exec) install targets may race against the
respective targets of the same name in one of the other multilibs of the
same library.

Issue (a) and (b) are addressed in a followup ((a) concerns also a
src/combined tree); AFAICS only (a) has the chance to cause silent
breakage in the install tree.

There are a couple of ways to fix (c):

0) Forbid parallel install, in documentation or in code.  Since install
is mostly IO-bound (at least with the multi-file installations present
in the tree and in Automake 1.11), this is not a big problem.  OTOH,
users might have bigger iron still making it worthwhile; automated build
systems might just pass a parallel make from higher up,

1) in Automake: ensure that install-multi is not run concurrently with
install-exec-am or install-data-am or a prerequisite of these,

2) in all multilib directories: ensure that no file is ever installed
twice to the same location from a multilib directory.  This would
require going through all multilibbed directories, identify all files
installed in a common location (typically headers), and let only one of
the multilibs actually install them.

My gut feeling is that the latter is a maintenance hassle.  So below is
a proposed patch for Automake that implements (1).  The patch changes
semantics of multilibbed directories a bit: while formerly,
  cd $target/libFOO && make install-exec

would have invoked 'make install' in each $target$MULTISUBDIR/libFOO,
now install-exec remains local to the $target/libFOO subtree.  This
might have implications for non-GCC packages using multilibs.  GCC
itself does not seem to care much about the 'install' vs. 'install-data'
vs. 'install-exec' issues.

As an alternative solution, it would be possible to let each of
install-{data,exec} have multilib recursion semantics and let the
toplevel serialize each wrt. $MULTISUBDIRs.  That would require changing
config-ml.in, and haveing install-{data,exec}-multi, and a few ripples
throughout GCC and src.  Not implemented yet, but I can do that if it is
preferable.

My question for gcc-patches is whether you agree with this conclusion.
For actual inclusion in GCC, I think it's fine to wait until the next
time a new Automake release is used in GCC.

For now, I have commented out the (nonportable) loop to expose the race
condition in the Automake test suite, as it's currently far too unlikely
thus needs very long to ever trigger on systems I test.  Time for
$expensive_tests I guess.

Thanks,
Ralf

(Adding Gary to THANKS not shown)

    Serialize multilib install rule, for concurrent same-file installs.
    
    * lib/am/multilib.am: Typo fixes in comments.
    (install-exec-am): Do not depend on install-multi.
    * automake.in (handle_install): New transform %?MULTI%, for
    toplevel multilib dirs.
    * lib/am/install.am (install-am) [MULTI]: Invoke install-multi
    in a sub-make, for ordering constraints.
    * tests/multlib.test: Actually install a few headers and
    libraries, so that a loop over `make -j install' has a chance of
    eventually failing.
    * NEWS, THANKS: Update.
    Reported against GCC by Gary Funck in
    <http://gcc.gnu.org/bugzilla/show_bug.cgi?id=42980>.

diff --git a/NEWS b/NEWS
index 4f8f8bb..bafe37e 100644
--- a/NEWS
+++ b/NEWS
@@ -53,6 +53,13 @@ Bugs fixed in 1.11a:
   - Several scripts as well as the parallel-tests testsuite driver now
     exit with the right exit status upon receiving a signal.
 
+  - The multilib `install' helper rule `install-multi' is now run off of
+    the internal `install-am' rule rather than `install-exec-am', to avoid
+    race conditions with parallel install due to mutliple versions of the
+    same data files being installed by prerequisites of install-data-am.
+    This changes implies that `make install-exec install-data' will not
+    descend sibling multilib directories any more, you need to use `make
+    install'.
 
 New in 1.11:
 
diff --git a/automake.in b/automake.in
index c0bf146..f7b5e07 100644
--- a/automake.in
+++ b/automake.in
@@ -4663,7 +4663,8 @@ sub handle_install ()
                             : ''),
      'installdirs-local' => (user_phony_rule 'installdirs-local'
                             ? ' installdirs-local' : ''),
-     am__installdirs => variable_value ('am__installdirs') || '');
+     am__installdirs     => variable_value ('am__installdirs') || '',
+     MULTI               => $seen_multilib && $relative_dir eq '.');
 }
 
 
diff --git a/lib/am/install.am b/lib/am/install.am
index d261070..8531e91 100644
--- a/lib/am/install.am
+++ b/lib/am/install.am
@@ -1,6 +1,6 @@
 ## automake - create Makefile.in from Makefile.am
-## Copyright (C) 2001, 2002, 2003, 2004, 2006, 2008, 2009  Free Software
-## Foundation, Inc.
+## Copyright (C) 2001, 2002, 2003, 2004, 2006, 2008, 2009, 2010 Free
+## Software Foundation, Inc.
 
 ## This program is free software; you can redistribute it and/or modify
 ## it under the terms of the GNU General Public License as published by
@@ -70,7 +70,9 @@ endif %?maybe_BUILT_SOURCES%
 .MAKE .PHONY: install-am
 install-am: all-am
        @$(MAKE) $(AM_MAKEFLAGS) install-exec-am install-data-am
-
+if %?MULTI%
+       @$(MAKE) $(AM_MAKEFLAGS) install-multi
+endif
 
 .PHONY: installcheck
 ?SUBDIRS?installcheck: installcheck-recursive
diff --git a/lib/am/multilib.am b/lib/am/multilib.am
index 6897b5d..2da76b4 100644
--- a/lib/am/multilib.am
+++ b/lib/am/multilib.am
@@ -1,5 +1,5 @@
 ## automake - create Makefile.in from Makefile.am
-## Copyright (C) 1998, 2001, 2003, 2004 Free Software Foundation, Inc.
+## Copyright (C) 1998, 2001, 2003, 2004, 2010 Free Software Foundation, Inc.
 
 ## This program is free software; you can redistribute it and/or modify
 ## it under the terms of the GNU General Public License as published by
@@ -29,6 +29,7 @@ all-multi:
        $(MULTIDO) $(AM_MAKEFLAGS) DO=all multi-do # $(MAKE)
 install-multi:
        $(MULTIDO) $(AM_MAKEFLAGS) DO=install multi-do # $(MAKE)
+## No uninstall rule?
 
 .MAKE .PHONY: all-multi install-multi
 
@@ -44,9 +45,6 @@ maintainer-clean-multi:
 
 .MAKE .PHONY: mostlyclean-multi clean-multi distclean-multi 
maintainer-clean-multi
 
-install-exec-am: install-multi
-## No uninstall rule?
-
 
 ## These cleaning rules are recursive.  They should not be
 ## registered as dependencies of *-am rules.  For instance
diff --git a/tests/multlib.test b/tests/multlib.test
index 65b3fd4..2e6e848 100755
--- a/tests/multlib.test
+++ b/tests/multlib.test
@@ -1,5 +1,5 @@
 #! /bin/sh
-# Copyright (C) 2003, 2004, 2007  Free Software Foundation, Inc.
+# Copyright (C) 2003, 2004, 2007, 2010 Free Software Foundation, Inc.
 #
 # This program is free software; you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -73,11 +73,23 @@ AC_OUTPUT
 END
 
 cat >libfoo/Makefile.am <<'END'
-noinst_LIBRARIES = libfoo.a
+lib_LIBRARIES = libfoo.a libfoo2.a libfoo3.a libfoo4.a libfoo5.a libfoo6.a
 libfoo_a_SOURCES = foo.c
+libfoo2_a_SOURCES = foo.c
+libfoo3_a_SOURCES = foo.c
+libfoo4_a_SOURCES = foo.c
+libfoo5_a_SOURCES = foo.c
+libfoo6_a_SOURCES = foo.c
+include_HEADERS = foo.h foo2.h foo3.h foo4.h foo5.h foo6.h
 END
 
 : > libfoo/foo.c
+: > libfoo/foo.h
+: > libfoo/foo2.h
+: > libfoo/foo3.h
+: > libfoo/foo4.h
+: > libfoo/foo5.h
+: > libfoo/foo6.h
 
 mkdir libbar
 
@@ -123,11 +135,12 @@ $AUTOCONF
 $AUTOMAKE --add-missing
 cd ..
 
+inst=`pwd`/inst
 
 # Check VPATH builds
 mkdir build
 cd build
-../configure --enable-multilib CC=mycc
+../configure --enable-multilib CC=mycc --prefix="$inst"
 $MAKE
 $MAKE install
 $MAKE distcleancheck
@@ -135,5 +148,10 @@ $MAKE distcleancheck
 # Check standard builds.
 cd ..
 # Why to I have to specify --with-target-subdir?
-./configure --enable-multilib --with-target-subdir=. CC=mycc
+./configure --enable-multilib --with-target-subdir=. CC=mycc --prefix="$inst"
 $MAKE
+
+# To actually test for the install race, I need something like this:
+#for i in `seq 2000`; do
+#  $MAKE -j install || { echo "failed at $i" >&2; Exit 1; }
+#done