Re: Problem loading numpy and other on a venv in a guix system

[Wojtek Kosior]

Hello,


> Hi Wojtek,
> 
> thanks for your help, indeed setting LD_LIBRARY_PATH solves the issue, 
> but it is less than ideal as I have to first
> check exactly the PATH to set, and if I update guix this will also 
> change I believe, still it is not so bad. I'd still like
> to understand why this happens in the first place.
> 
> Best regards,
> Mateus Rodrigues

I see.

I don't think I realize all the nuances of dynamic library loading
myself. But I can try to explain what I know :)

In C (also to some extent C++) world, libraries can be either linked
statically into a program (i.e. included in its executable) or linked
dynamically (i.e. so that they are found in the filesystem when program
gets executed). There are pros and cons to both approaches.

With dynamically-linked libraries (also called shared libraries), it is
possible to write a program so that it doesn't load them immediately
upon start. It is possible to instead have program's code call a C
library function that loads a shared library *by name*, together with
other shared libraries it depends on. This is what the Python
interpreted process tries to do when told to load Numpy's C extension.

Under a typical GNU/Linux system, shared libraries are searched for in
some predefined locations (like `/lib/x86_64-linux-gnu/` which is where
`libz.so.1` resides in my Debian OS). AFAIK, these locations are
compiled into the C library and cannot be *changed*. However, they can
be *supplemented* by paths in the `LD_LIBRARY_PATH` variable which are
searched by the C library *before* the standard locations.

Now, Guix does things differently. It does not keep its libraries in
standard locations like `/lib`. It cannot - the assumption is that
files needed by programs are kept under `/gnu/store`. So, AFAIU, the C
library in Guix is *modified* not to assume any standard directories
for shared libraries. Instead, all programs have the exact locations of
their shared libraries specified directly in their executables. This is
possible with the help of a (perhaps a bit obscure) feature called
`RPATH`.

So we already know that Guix' own packages make up for the lack of
standard shared library directories. How about binaries like those
from PyPI that were not compiled for Guix? They don't have the RPATH
set and they expect libc to be able to resolve some of their
dependencies. In particular, Python C extensions devs used to be rather
safe to assume that zlib (a dependency of CPython) would be present and
resolvable in the user's operating system and need not be bundled with
the extension.

If we think about it, an analogous problem appears for many languages,
not just C. And it seems Guix solves it slightly differently there. For
example, Python library locations are specified in Guix-prepared
variable `GUIX_PYTHONPATH`. I believe that works more reliably than C's
`RPATH` (although I had some issues with loading of Python libs, too).

Back to your problem - symlinks to shared libraries are also placed in
the `lib/` subdirectory of a Guix profile. For example, after doing

    guix install zlib

I got a `libz.so.1` inside `~/.guix-profile/lib`. Perhaps this will
prove helpful to you? :)

Wojtek

-- (sig_start)
website: https://koszko.org/koszko.html
PGP: https://koszko.org/key.gpg
fingerprint: E972 7060 E3C5 637C 8A4F  4B42 4BC5 221C 5A79 FD1A

Meet Kraków saints!           #39: blessed Narcyz Turchan
Poznaj świętych krakowskich!  #39: błogosławiony Narcyz Turchan
https://pl.wikipedia.org/wiki/Narcyz_Turchan
-- (sig_end)


On Wed, 30 Nov 2022 09:42:34 +0000
mprodrigues@posteo.net wrote:

> Hi Wojtek,
> 
> thanks for your help, indeed setting LD_LIBRARY_PATH solves the issue, 
> but it is less than ideal as I have to first
> check exactly the PATH to set, and if I update guix this will also 
> change I believe, still it is not so bad. I'd still like
> to understand why this happens in the first place.
> 
> Best regards,
> Mateus Rodrigues
> 
> On 29.11.2022 19:33, Wojtek Kosior wrote:
> > Hi Mateus!
> > 
> > Indeed, when using virtualenv's numpy, this
> > 
> >     python3 -c 'import numpy'
> > 
> > fails with the error you posted. I seem to have been able to make numpy
> > happy by instead running Python like
> > 
> > 
> > LD_LIBRARY_PATH=/gnu/store/8qv5kb2fgm4c3bf70zcg9l6hkf3qzpw9-zlib-1.2.11/lib/
> > python3 -c 'import numpy'
> > 
> > If you use the `--container` and `--emulate-fhs` options to
> > `guix shell`, it is also possible to instead do
> > 
> >     LD_LIBRARY_PATH=/usr/lib/ python3 -c 'import numpy'
> > 
> > The `libz` guix package still needs to be included among the packages
> > passed to `guix shell` in this last case.
> > 
> > Also, given you're using virtualenv, you might find the `--network`
> > option to `guix shell` useful :)
> > 
> > I hope there isn't any dumb mistake in my solution... Good luck :)
> > 
> > Wojtek
> > 
> > -- (sig_start)
> > website: https://koszko.org/koszko.html
> > PGP: https://koszko.org/key.gpg
> > fingerprint: E972 7060 E3C5 637C 8A4F  4B42 4BC5 221C 5A79 FD1A
> > 
> > Meet Kraków saints!           #27: blessed Maria Klemensa Staszewska
> > Poznaj świętych krakowskich!  #27: błogosławiona Maria Klemensa 
> > Staszewska
> > https://pl.wikipedia.org/wiki/Maria_Klemensa_Staszewska
> > -- (sig_end)
> > 
> > 
> > On Tue, 29 Nov 2022 16:42:14 +0000
> > mprodrigues@posteo.net wrote:
> >   
> >> Hello,
> >> 
> >> I've been using a guix shell with a couple of python packages and it
> >> works fine,
> >> however I'd also like to keep the option of using a virtualenv if
> >> possible, the
> >> thing is I'm having problem running modules like numpy, that depend on
> >> non
> >> python dependencies. The error I get when I try to load numpy in a 
> >> venv
> >> is:
> >> 
> >> Importing the numpy C-extensions failed. This error can happen for 
> >> many
> >> reasons,
> >> often due to issues with your setup or how NumPy was installed.
> >> 
> >> and in the end:
> >> 
> >> Original error was: libz.so.1: cannot open shared object file: No such
> >> file or
> >> directory
> >> 
> >> I have tried installing zlib directly, also tried to add my default
> >> profile libs
> >> path to the venv PATH, but to no avail until now.  How can I solve 
> >> this?
> >> Or
> >> maybe someone can point me in the direction that I have to take to
> >> properly
> >> debug this.
> >> 
> >> Best regards,
> >> Mateus Rodrigues
> >>

pgpfYfLMUFpuy.pgp
Description: OpenPGP digital signature