bug#39930: python-virtualenv produces a broken pip when network is available

[Jakub Kądziołka]

I am encountering a weird error:

| ~/tmp$ guix environment --pure --ad-hoc python python-virtualenv coreutils
| ~/tmp% pip3 --version
| pip 19.0.3 from 
/gnu/store/az4z48nrlyhrpkj2njs6xz3p0mj71wi7-profile/lib/python3.7/site-packages/pip
 (python 3.7)
| ~/tmp% virtualenv venv
| Using base prefix '/gnu/store/608bvypsh90c58apvd2cgg3m9l2pwjqn-python-3.7.4'
| New python executable in /home/kuba/tmp/venv/bin/python
| Installing setuptools, pip, wheel...
| done.
| ~/tmp% . venv/bin/activate
| (venv) ~/tmp% pip3 --version # or any other command
| Traceback (most recent call last):
|   File "/home/kuba/tmp/venv/bin/pip3", line 7, in <module>
|     from pip._internal.cli.main import main
| ModuleNotFoundError: No module named 'pip._internal.cli.main'

Note that unsetting PYTHONPATH helps:

| (venv) ~/tmp% PYTHONPATH= pip3 --version
| pip 20.0.2 from /home/kuba/tmp/venv/lib/python3.7/site-packages/pip (python 
3.7)

Similarily, omitting the python package from the environment also helps:

| ~/tmp$ guix environment --pure --ad-hoc python-virtualenv coreutils
| ~/tmp% virtualenv venv
| Using base prefix '/gnu/store/608bvypsh90c58apvd2cgg3m9l2pwjqn-python-3.7.4'
| New python executable in /home/kuba/tmp/venv/bin/python
| Installing setuptools, pip, wheel...
| done.
| ~/tmp% . venv/bin/activate
| (venv) ~/tmp% pip3 --version
| pip 20.0.2 from /home/kuba/tmp/venv/lib/python3.7/site-packages/pip (python 
3.7)

Interestingly, even with PYTHONPATH set, everything works in a container
started without network access:

| ~/tmp$ guix environment --container --ad-hoc python python-virtualenv 
coreutils
| kuba@gravity ~/tmp [env]$ virtualenv venv
| Using base prefix '/gnu/store/608bvypsh90c58apvd2cgg3m9l2pwjqn-python-3.7.4'
| New python executable in /home/kuba/tmp/venv/bin/python
| Installing setuptools, pip, wheel...
| done.
| kuba@gravity ~/tmp [env]$ . venv/bin/activate
| (venv) kuba@gravity ~/tmp [env]$ pip3 --version
| pip 19.0.3 from 
/gnu/store/az4z48nrlyhrpkj2njs6xz3p0mj71wi7-profile/lib/python3.7/site-packages/pip
 (python 3.7)

My guess as to what's happening: virtualenv tries to use a newer pip
than is provided by Guix. It then generates an entrypoint appropriate
for the pip version it chose and installed into the venv. Unfortunately,
the $PYTHONPATH set by Guix overrides virtualenv's mechanism for
directing Python to venv/lib/python3.7/site-packages. This makes an
entrypoint designed for pip 20.0.2 try to load pip 19.0.3. pip's main
function was relocated between releases, making this problematic.

I am not sure which part of that is wrong here, nor how to go about
fixing it. The problem would disappear temporarily when we merged
core-updates into master, but keeping up with Python's release schedule
doesn't sound feasible.

Note that while virtualenv doesn't change PYTHONPATH, it does unset
PYTHONHOME - maybe that's a more appropriate environment variable to use
for pointing Python to the primary modules directory?

While debugging this, I noticed that the virtualenv we ship is outdated.
I prepared a patchstack, which I will send later today, to update the
version to latest virtualenv. The behavior didn't change, apart from
slightly different output formatting of virtualenv.

Regards,
Jakub Kądziołka

signature.asc
Description: PGP signature