Re: Python, poetry, and pandas

[Sébastien Lerique]

Hi Phil and Simon,

Thanks both for the input and advice (not citing it all here).

First the details. Simon wrote:
> How did you installed poetry?  Via Guix or something else?
> Which python packages do you have in your profile?
> Is Guix System or Guix on foreign distro?

I'm using Guix System, and installed poetry via Guix. I have the 
following python packages installed: python, poetry, 
python-jupyterlab (from guix-science), and python-black. And 
guix-jupyter.

> > (I am aware that python-pandas is packaged for guix, but some
> > packages I will need down the road might not be; this project 
> > is
> > also in collaboration with non guix users, so I am looking for 
> > a
> > workflow which will work for them too.)
>
> What are these packages?

I don't know yet, but I suspect the situation might crop up. Phil 
pointed to the PyPi importer, and I haven't used it yet, but it 
does seem to make things simpler if I need to add a package to 
Guix.

Phil wrote:
> > ImportError: libstdc++.so.6: cannot open shared object file: No 
> > such
> > file or directory
>
> If it's only missing a reference to the underlying C++ library 
> you could
> install this library using Guix, or if you're using a foreign 
> distro point to
> the non-Guix version under /usr/lib which, at a guess, Poetry 
> may have
> picked up - but this is going to cause you more trouble down the 
> line
> than repackaging in Guix in my experience.

Yes, my experience with trying this for some Julia packages 
(GLMakie) convinced me not to try LD_LIBRARY_PATH hacks any more.

Now the broader view. Phil:
> Guix offers pretty much everything that virtualenvs offers, but 
> it does
> things a little differently - you'll need to look at 
> environments,
> profiles, manifests, and use of a guix.scm to recreate (IMHO) a 
> better
> experience in the end.

I just set up a simple guix.scm for my current project, which 
indeed does the job, and of course provides all the great 
guarantees of Guix! Given the number of python packages currently, 
and the existence of the PyPi importer, I'm pretty convinced it 
covers my personal needs.

Now my wish is to find ways to collaborate with others on this 
kind of project, who unfortunately often use macOS. I was 
exploring the possibility of Guix on Darwin, but the best way to 
do that looks like running Guix System inside Docker:

http://logs.guix.gnu.org/guix/2021-03-11.log#044004
https://lists.gnu.org/archive/html/guix-devel/2017-10/msg00111.html

So it seems that to collaborate with non-guix and non-GNU/Linux 
users, the options are 1) maintain a second environment management 
configuration (e.g. using poetry) to provide a stop-gap, or 2) 
convince colleagues to run Guix in Docker. The situation for 
Windows might be a bit better using the Linux Subsystem.

Thanks again all,
Sébastien

On 10 Mar 2021 at 19:02, Phil <phil@beadling.co.uk> wrote:

> Hi Sébastien,
>
> Sébastien Lerique writes:
>
> > Dear all,
> >
> > (I am aware that python-pandas is packaged for guix, but some 
> > packages
> > I will need down the road might not be; this project is also in
> > collaboration with non guix users, so I am looking for a 
> > workflow
> > which will work for them too.)
>
> Under the covers Poetry is just using regular Python virtualenvs 
> I think.
> These don't work well with Guix for a number of reasons - but in 
> my
> previous attempts to use them I found that Guix's use of 
> PYTHONPATH
> meant that virtualenvs where never isolated from packages 
> installed by
> Guix.
>
> Guix offers pretty much everything that virtualenvs offers, but 
> it does
> things a little differently - you'll need to look at 
> environments,
> profiles, manifests, and use of a guix.scm to recreate (IMHO) a 
> better
> experience in the end.
>
> When I started using Guix I went to some lengths to harmonize
> virtualenvs and Guix - and others have made more substantial 
> efforts
> than I did - but you end up creating as many problems as you 
> fix:
> https://lists.gnu.org/archive/html/help-guix/2020-11/msg00237.html
>
> If you search the mail archive a couple of other people have 
> asked
> similar questions to do with poetry, pyenv, pip, etc - they're 
> all
> interesting reading and easy to grep with those keywords:
> https://lists.gnu.org/archive/html/help-guix/
>
> The advice I was given at the time was don't mix package 
> managers, and
> in hindsight I'm glad repackaged what I needed under Guix.
>
> I've packaged quite a few python libraries now, and most are 
> pretty
> trivial to do - if they are in PyPi you can just use this:
>
> guix import pypi <package_name>
>
> If you need a later version than Guix provides (eg pandas), my 
> advice is
> to use the ability to "inherit" the current definition in Guix 
> and tweak
> the version.  For Pandas I did this for a more recent version I 
> needed
> and all I had to do was disable 2 unit tests (which failed for a
> non-Guix reason).
>
> Apologies - not exactly the answer you want (I have a stab at 
> that
> below), but it's worth considering moving to Guix packaging 
> exclusively
> to keep your life simple.
>
>
> >  File
> > "/home/sl/.cache/pypoetry/virtualenvs/poetry-test-vOWX4_Vr-py3.8/lib/python3.8/site-packages/pandas/core/window/ewm.py",
> > line 9, in <module>
> >    import pandas._libs.window.aggregations as 
> >    window_aggregations
> > ImportError: libstdc++.so.6: cannot open shared object file: No 
> > such
> > file or directory
>
> If it's only missing a reference to the underlying C++ library 
> you could
> install this library using Guix, or if you're using a foreign 
> distro point to
> the non-Guix version under /usr/lib which, at a guess, Poetry 
> may have
> picked up - but this is going to cause you more trouble down the 
> line
> than repackaging in Guix in my experience.