"no module named posixpath" with virtualenv 20

[jaraco]

In pypa/setuptools#1998, I observed a regression in the Setuptools test suite when virtualenv 20 is used to create the test env.

Thusfar, I've only been able to replicate the issue in the Setuptools test suite. I'm working now on creating a more selective reproducer.

[jaraco]

I've been able to pare the problem down:

First, create two envs on select versions of virtualenv:

draft $ pip-run -q 'virtualenv<20' -- -m virtualenv --python python2 env16
draft $ pip-run -q 'virtualenv>20' -- -m virtualenv --python python2 env20

Then, create a pkgs to put on PYTHONPATH:

draft $ mkdir pkgs
draft $ ln -s ~/p/pypa/setuptools/setuptools/site-patch.py pkgs/site.py

Note that this pkgs directory is comparable to what setup.py develop --install-dir pkgs or pip install -e <dir> --target pkgs would install by way of install_site_py.

Now observe that the environment starts up fine on a virtualenv created for virtualenv 16 but not 20.

draft $ env PYTHONPATH=pkgs env16/bin/python -c "import os"
draft $ env PYTHONPATH=pkgs env20/bin/python -c "import os"
Traceback (most recent call last):
  File "pkgs/site.py", line 73, in <module>
    __boot()
  File "pkgs/site.py", line 3, in __boot
    import os
  File "/Users/jaraco/draft/env20/lib/python2.7/os.py", line 49, in <module>
    import posixpath as path
ImportError: No module named posixpath

[gaborbernat]

Wait, what? You're replacing site.py with something else? That's just not supported at the moment. Why would anyone want to do this, is that really a smart Idea?

[jaraco]

I know, right? It's filthy and ugly, and apparently has been around for quite some time (pypa/setuptools@8c89c94, pypa/setuptools@3df2aab).

Why would anyone want to do this, is that really a smart Idea?

I'm not sure. There's some background in the second link above.

[gaborbernat]

virtualenv 20 is just by implementation incompatible with this approach, so not sure what we can do 🤷‍♀️

[gaborbernat]

So sounds like to me won't support. I believe all such use cases that want this should move to pth files. 🤔

[gaborbernat]

Note: this version includes a hacked 'site.py' to support processing .pth files in directories that come before site-packages on sys.path. However, because of its placement, it should only come into play when a user puts the setuptools .egg file directly on PYTHONPATH, so it doesn't affect "virtual" or "root" installations. It's strictly to provide support for luddites who refuse to give up their existing non-root PYTHONPATH setup unless you pry it from their cold, dead hands. :)

That's a commit from 2005. I think with 2020 on our hands we can safely mark that unsupported for whoever refuse to give up their existing non-root PYTHONPATH setup unless you pry it from their cold, dead hands.. If they want that behaviour use old virtualenv; or use old setuptools. I'd remove this feature from core setuptools.

[jaraco]

I'd remove this feature from core setuptools.

This may be the best course of action.

Regardless, I think there's a more serious issue at play here. Somehow environments created by venv aren't subject to this issue, only virtualenv 20 is. That signals to me that the new virtualenv might be doing something more invasive and potentially problemmatic to other use cases.

I explored further and discovered that the issue does not happen on recent versions of Python, only on Python 2.7. The following example illustrates the issue without involving setuptools, only using an empty site.py.

draft $ mkdir pkgs
draft $ touch pkgs/site.py
draft $ pip-run -q 'virtualenv>20' -- -m virtualenv env20
created virtual environment in 426ms CPython3Posix(dest=/Users/jaraco/draft/env20, clear=False, global=False) with seeder FromAppData pip=latest setuptools=latest wheel=latest app_data_dir=/Users/jaraco/Library/Application Support/virtualenv/seed-v1 via=copy
draft $ env PYTHONPATH=pkgs env20/bin/python -c "import os"
draft $ pip-run -q 'virtualenv>20' -- -m virtualenv --python python2 env20-py27
created virtual environment in 340ms CPython2Posix(dest=/Users/jaraco/draft/env20-py27, clear=False, global=False) with seeder FromAppData pip=latest setuptools=latest wheel=latest app_data_dir=/Users/jaraco/Library/Application Support/virtualenv/seed-v1 via=copy
draft $ env PYTHONPATH=pkgs env20-py27/bin/python -c "import os"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/jaraco/draft/env20-py27/lib/python2.7/os.py", line 49, in <module>
    import posixpath as path
ImportError: No module named posixpath

It seems plausible to me that this limitation may impact other use-cases, though it does also seem unlikely.

[gaborbernat]

I could have told you this upfront. We inject our own custom site.py for python 2 to fix up sys.path paths, and then that calls the system site.py. If you overwrite that, things will not work; see https://github.com/pypa/virtualenv/blob/master/src/virtualenv/create/via_global_ref/builtin/python2/site.py

This was no different with the old virtualenv; but we linked more of stdlib in rather than fix it via site.py; hence may have worked more often. But note if you would import any other package from stdlib within the newly generated site.py than what's here https://github.com/pypa/virtualenv/blob/legacy/virtualenv.py#L158-L178, you would have seen the same failure.

Even if you don't do that if you overwrite the site.py we no longer can apply fixes within the injected site.py, that we must to end up with a working virtual environment: see https://github.com/pypa/virtualenv/blob/legacy/virtualenv_embedded/site.py#L734-L741. If these lines are not run within the new generated the site.py you would end up with a broken virtualenv even pre 20.

Long story short on Python 2 any kind of overwrite of site.py throws you straight up in unsupported land.

The only type of site.py modification of site.py is extension; which usually means pth files (unless you want to append to the site.py file).

[boegel]

I was hitting this same problem with tox when virtualenv 20.x was installed, but it manifests weirdly (-v exposes the issue though):

$ tox -c tox.ini -e py27 -v
using tox.ini: /Volumes/work/vsc-base/tox.ini (pid 26625)
using tox-3.14.6 from /Users/kehoste/.pypkgs/lib/python2.7/site-packages/tox/__init__.pyc (pid 26625)
py27 cannot reuse: no previous config /Volumes/work/vsc-base/.tox/py27/.tox-config1
py27 create: /Volumes/work/vsc-base/.tox/py27
[26627] /Volumes/work/vsc-base/.tox$ /System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python -m virtualenv --no-download --python /System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python py27 >py27/log/py27-0.log
u'exit code 1' cmd '/Volumes/work/vsc-base/.tox/py27/bin/python' out '' err 'Traceback (most recent call last):\n  File "/Users/kehoste/.pypkgs/lib/python2.7/site-packages/site.py", line 73, in <module>\n    __boot()\n  File "/Users/kehoste/.pypkgs/lib/python2.7/site-packages/site.py", line 3, in __boot\n    import os\n  File "/Volumes/work/vsc-base/.tox/py27/lib/python2.7/os.py", line 49, in <module>\n    import posixpath as path\nImportError: No module named posixpath\n'
_______________________________________________________________ summary _______________________________________________________________
  py27: commands succeeded
  congratulations :)
Traceback (most recent call last):
  File "/Users/kehoste/.pypkgs/bin/tox", line 8, in <module>
    sys.exit(cmdline())
  File "/Users/kehoste/.pypkgs/lib/python2.7/site-packages/tox/session/__init__.py", line 44, in cmdline
    main(args)
  File "/Users/kehoste/.pypkgs/lib/python2.7/site-packages/tox/session/__init__.py", line 68, in main
    exit_code = session.runcommand()
  File "/Users/kehoste/.pypkgs/lib/python2.7/site-packages/tox/session/__init__.py", line 194, in runcommand
    return self.subcommand_test()
  File "/Users/kehoste/.pypkgs/lib/python2.7/site-packages/tox/session/__init__.py", line 222, in subcommand_test
    run_sequential(self.config, self.venv_dict)
  File "/Users/kehoste/.pypkgs/lib/python2.7/site-packages/tox/session/commands/run/sequential.py", line 9, in run_sequential
    if venv.setupenv():
  File "/Users/kehoste/.pypkgs/lib/python2.7/site-packages/tox/venv.py", line 621, in setupenv
    envlog.set_python_info(command_path)
  File "/Users/kehoste/.pypkgs/lib/python2.7/site-packages/tox/logs/env.py", line 18, in set_python_info
    answer["executable"] = python_executable
TypeError: 'NoneType' object does not support item assignment

How lovely of tox to claim all tests have passed when it couldn't even find the python executable to test with.

Downgrading to virtual<20 fixed the issue for me.

I wanted to mention this here since it's the same issue, but I should probably (also) report this to tox?

[gaborbernat]

@boegel is that with latest virtualenv? Should not be happening.