OMPI workaround (was: Problem with LT 2.1a)

[Jeff Squyres]

On 9/6/06 11:10 AM, "Ralf Wildenhues" <address@hidden> wrote:

> First let me try to point out a workaround for your specific situation:
> 
> Since mthca.so is Linux-specific, we can rely on having dlopen available
> for the workaround.  Since I assume you would rather not have to modify
> the part (3), let's modify (2): Before dlopen("mthca.so") inside
> libibverbs, let ompi_plugin do an explicit dlopen("libibverbs.so",
> RTLD_GLOBAL).  This should cost almost nothing (no system calls
> involved, since libibverbs.so is already opened), and should make the
> set of symbols available for the subsequent mthca.so.
> 
> Do I understand correctly that since you do not use the lt_dl*search*
> functions anyway, adding this functionality to opal would be rather
> painless?

(changed the subject to differentiate this reply from the rest of the
thread)

Actually, I wouldn't put it in opal, but rather in the ompi_plugin directly,
because at least currently, it's only a problem for one specific OMPI plugin
(specifically, the OpenIB BTL plugin).

However, the problem here is that this depends on the C code knowing exactly
where the library is.  This unfortunately matters for at least one important
scenario: the OFED installer (Open Fabrics Enterprise Distribution --
currently, an InfiniBand software stack installer) builds RPMs for all of
its software (including OMPI) in a temporary tree.  This includes packages
that depend on each other (e.g., OMPI depends on libibverbs).  Hence, the
libibverbs RPM is created and its [staging] installation tree is simply not
deleted.  Other packages that depend on the libibverbs are expected to use
the [staging] installation tree.

Once all of OFED is built, all the RPM build/install trees are deleted.
Installing the RPMs then installs the software into the "real" locations
(and therefore dependencies change).

Currently, OMPI skirts this problem because OMPI is only built once the rest
of OFED is installed in their final locations.  I am currently tasked with
making OMPI be able to be built when all the rest of the OFED software is
built (not after the fact, like it is now).  Ralf W. and I have had
extensive discussions about this and I have not yet had time to try his
suggestion (link directly to libibverbs.la in the staging installation tree
instead of "-L/staging/install/tree -libverbs").

But the point remains here that if we go the dlopen() route, then the C code
needs to be aware of where libibverbs is -- and at least in this scenario,
it could be in one of two places (and at build time, I won't know that).
The C code could simply traverse LD_LIBRARY_PATH and look for libibverbs,
but that then subverts the ld.so mechanisms and also generally seems icky.

-- 
Jeff Squyres
Server Virtualization Business Unit
Cisco Systems