lt_dlopenext() doesn't find libraries installed into a prefix on Solaris

[Török Edwin]

Hi,

When using libtool 2.2.6, and ltdl on Solaris 10 lt_dlopenext() won't
find a library that has been installed
into a prefix that is not on system's search path. This works on Linux
though.

Example:
./configure --prefix=/usr/local/clamav-0.95rc2
make && make install

libclamav.so does this:
lt_dlinit()
lt_dlopenext("libclamunrar_iface");

On neither systems is the .la file searched for in the install prefix,
however on Linux the .so *is* found in the prefix
when dlopening is attempted here:

libltdl/loaders/dlopen.c:194:
module = dlopen (filename, module_flags);

filename is simply "libclamunrar_iface.so", and as you can see from the
strace on Linux [1], it *does* open libclamunrar_iface.so from the prefix.
However on Solaris this doesn't happen.

According to the dlopen manpage (on Linux) it should search for the
library in the runpath/rpath,
here's what readelf says about those hardcoded paths in the binaries:

I looked at the binaries, and on Linux I have this (when I build with
--prefix=$HOME/pfx)
$ eu-readelf -a pfx/bin/clamscan|grep R*PATH
  RPATH             Library rpath: [/home/edwin/pfx/lib]
$ eu-readelf -a pfx/lib/libclamav.so|grep R*PATH

On Solaris (when I build with --prefix=$HOME/pfx)
$ eu-readelf -a clamscan|grep R*PATH
  RUNPATH           Library runpath:
[/export/home/edwin/pfx/lib:/usr/sfw/lib]
  RPATH             Library rpath: [/export/home/edwin/pfx/lib:/usr/sfw/lib]
$ eu-readelf -a libclamav.so|grep R*PATH
  RUNPATH           Library runpath: [/usr/sfw/lib]
  RPATH             Library rpath: [/usr/sfw/lib]

So the executable contains prefix/lib in the runpath (but dlopen doesn't
search there), and the .so that uses libtool
to dlopen (libclamav.so) doesn't have a runpath set in Linux (does it
inherit the one from the executable clamscan?),
and the one in Solaris has a runpath set (does it overwrite the runpath
of the executable clamscan?)

For full details see https://wwws.clamav.net/bugzilla/show_bug.cgi?id=1476]
[1] strace on Linux: https://wwws.clamav.net/bugzilla/attachment.cgi?id=967
[2] truss trace on Solaris:
https://wwws.clamav.net/bugzilla/attachment.cgi?id=966

As a workaround on Solaris LD_LIBRARY_PATH can be set, but I think that
using libtool should "just work"
without setting LD_LIBRARY_PATH.

Best regards,
--Edwin