Various race condition in install mode.

[Sam Varshavchik]

I've almost tracked down the source of random link-install failures that 
unexpectedly blows up a long build cycle.

I'm using libtool 1.5.6 with automake 1.8.3 on Linux.  I'm building a fairly 
extensive app that installs a bunch of shared libraries and executables.  
And it uses a bunch of convenience libraries along the way.

The build cycle runs on an SMP box, and I use the -j option with GNU make to 
build stuff in parallel.  gmake's -j option causes gmake to kick off 
multiple build commnds as long as, according to Makefile rules, they don't 
depend on each other in any way.

I believe that I see at least one, and possibly two problems that result 
from this, as well as a third issue that needs to be clarified in libtool's 
documentation:

1)

One problem seems to be related to the following code fragment in the 
libtool script:

        if test "$mode" = relink; then
	  $run eval '(cd $output_objdir && $rm ${realname}U && $mv $realname 
${realname}U)' || exit $?
        fi

As far as I can figure this out, under certain conditions, during an install 
cycle (I think), libtool temporarily renames a shared library 
libname.so.a.b.c as libname.so.a.b.cU, then does something.  I also see 
this, elsewhere in libtool:

        # Restore the uninstalled library and exit
        if test "$mode" = relink; then
	  $run eval '(cd $output_objdir && $rm ${realname}T && $mv $realname 
${realname}T && $mv "$realname"U $realname)' || exit $?
          exit $EXIT_SUCCESS
        fi

        # Create links to the real library.
        for linkname in $linknames; do
          if test "$realname" != "$linkname"; then
	    $show "(cd $output_objdir && $rm $linkname && $LN_S $realname 
$linkname)"
	    $run eval '(cd $output_objdir && $rm $linkname && $LN_S 
$realname $linkname)' || exit $?
          fi
        done

So there's some switcheroo going on here, with the shared library files 
bouncing back and forth.

Now, an automake-generated Makefile will have multiple targets that are 
dependend on the 'make install' target (that's simplyfing things a bit, but 
that's the general idea).

The bottom line is that gmake is going to kick off multiple 'install' 
targets at the same time, because as far as automake knows, the install 
action for various binaries and libraries don't have any dependencies on 
each other.  An install just copies a file, and automake-generated makefile 
doesn't think that one file (be it a binary or a library) needs to have 
another file installed first.

So, you're going to have multiple invocations of libtool, in install mode, 
occuring at the same time.  And each instance is going to monkey around with 
the shared library files, renaming them back and forth.  I think that the 
race condition problem here should now appear to be quite obvious.

There used to be a similar problem with autoconf -- where config.status used 
scratch files with fixed names, and under certain conditions an 
automake-generated makefile could end up running multiple instance of 
config.status at the same time, so they ended up scribbling over each 
other's temporary files.  autoconf eventually got fixed by embedding the 
process id into temporary file names.

It looks like some form of locking is needed in install mode, so that 
multiple libtool installs are not going to stomp over each other's shared 
libraries.

One postscriptum to add is that I do not actually specify the -j option to 
gmake explicitly, I place the -j option in the MAKEFLAGS environment 
variable, so that each instance of a recursively-invoked gmake is going to 
run parallel stuff.

2)

Here's an automake-generate rule that depends on the overall 'make install 
target':

install-exec-am: install-pkglibLTLIBRARIES install-pkglibexecPROGRAMS \
       install-pkglibexecSCRIPTS install-sbinPROGRAMS \
       install-sbinSCRIPTS

As we see, there are two sub-targets here of interest: 
install-pkglibLTLIBRARIES and install-pkglibexecPROGRAMS.  The first one 
installs the shared libraries, the second one installs binaries that 
potentially link against the shared libraries.  Neither target depends on 
each other, they only both depend on their parent, the install-exec-am 
target.  Since install-pkglibexecPROGRAMS does not depend on 
install-pkglibLTLIBRARIES, gmake doesn't see any reason why it can't run 
these targets in parallel.  Conceivably gmake could end up installing one of 
the binaries before it finishes installing all the libraries.

> From researching the first issue, I understand that libtool in certain 
situations, will do linking during the install mode.  I haven't yet 
confirmed whether libtool tries to link against the libraries in their 
final, installation directory that it expects to be there already, or it 
will use the copy in the .libs subdirectory. If, as I suspect, it's linking 
against the libraries in their final install directories, they may not be 
there yet if automake ends up running the install target for one of the 
binaries before it finishes running the install targets for the libraries.

The reason why I suspect libtool is trying to link against libraries in 
their final installation directory is based on the third issue:

3)

> From Makefile.am:

pkglib_LTLIBRARIES=libcourierauthcommon.la libcourierauth.la $(modules)


Then I have also:

libcourierauthcommon_la_DEPENDENCIES=$(commonlibs) libcourierauth.la
libcourierauthcommon_la_LDFLAGS=libcourierauth.la [ more stuff here ]

The first thing make install does is this:

/bin/sh ./libtool --mode=install /usr/bin/install -c  
'libcourierauthcommon.la' 
'/tmp/x/usr/lib64/courier-authlib/libcourierauthcommon.la'
libtool: install: warning: relinking `libcourierauthcommon.la'
(cd /home/mrsam/rpm/courier-authlib-0.50-1/BUILD/courier-authlib-0.50; 
/bin/sh ./libtool  --quiet --mode=relink gcc -I/usr/include/mysql 
-I/usr/include -O2 -g -pipe -Wall -I.. -I./.. -o libcourierauthcommon.la 
-rpath /usr/lib64/courier-authlib libcourierauth.la -Wl,--whole-archive 
-Wl,userdb/libshuserdb.a -Wl,numlib/libshnumlib.a 
-Wl,random128/libshrandom128.a -Wl,gdbmobj/libshgdbmobj.a -Wl,libshauth.a 
-Wl,rfc822/libshencode.a -Wl,libhmac/libshhmac.a -Wl,md5/libshmd5.a 
-Wl,sha1/libshsha1.a -Wl,--no-whole-archive -lcrypt -lgdbm -ldb 
-inst-prefix-dir /tmp/x)

Which results in:

/usr/bin/ld: cannot find -lcourierauth

'make install' goes through the libraries in their listed order, and tries 
to install libcourierauthcommon first, which fails because libtool relinks 
against libcourierauth.la, which hasn't been installed yet.

The first library has a dependency on the second one, but automake really 
doesn't think that it should install the second library before the first 
one; that's really a dependency that's introduced by libtool.

Therefore, it needs to be documented that with libtool, the order of the 
listed libraries in _LTLIBRARIES should agree with any dependencies they 
have between themselves.

pgprKdT75bV7c.pgp
Description: PGP signature