Unable to run pip after easy_install -U pip with latest setuptools

[psybuzz]

Hello,

Today, I ran easy_install -U pip on a fresh virtualenv with setuptools 47.3.0 and running the pip command no longer works.

Repro steps (trimmed)

$ virtualenv --system-site-packages ~/piptest -p python3
$ source ~/piptest/bin/activate
(piptest) $ pip list
Package               Version
--------------------- -------------------
packaging             20.3
pip                   20.1.1
setuptools            47.3.0
virtualenv            16.7.7
wheel                 0.34.2

(piptest) $ easy_install -U pip
WARNING: The easy_install command is deprecated and will be removed in a future version.
Searching for pip
Reading https://pypi.org/simple/pip/
Downloading https://files.pythonhosted.org/packages/fe/3b/0fc5e63eb277d5a50a95ce5c896f742ef243be27382303a4a44dd0197e29/pip-20.2b1-py2.py3-none-any.whl#sha256=b4e230e2b8ece18c5a19b818f3c20a8d4eeac8172962779fd9898d7c4ceb1636
Best match: pip 20.2b1
Processing pip-20.2b1-py2.py3-none-any.whl
Installing pip-20.2b1-py2.py3-none-any.whl to ~/piptest/lib/python3.7/site-packages
Adding pip 20.2b1 to easy-install.pth file
Installing pip script to ~/piptest/bin
Installing pip3 script to ~/piptest/bin
Installing pip3.8 script to ~/piptest/bin

Installed ~/piptest/lib/python3.7/site-packages/pip-20.2b1-py3.7.egg

(piptest) $ pip
Traceback (most recent call last):
  File "~/piptest/bin/pip", line 30, in <module>
    sys.exit(load_entry_point('pip==20.2b1', 'console_scripts', 'pip')())
  File "~/piptest/lib/python3.7/site-packages/pkg_resources/__init__.py", line 490, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "~/piptest/lib/python3.7/site-packages/pkg_resources/__init__.py", line 482, in get_distribution
    dist = get_provider(dist)
  File "~/piptest/lib/python3.7/site-packages/pkg_resources/__init__.py", line 358, in get_provider
    return working_set.find(moduleOrReq) or require(str(moduleOrReq))[0]
  File "~/piptest/lib/python3.7/site-packages/pkg_resources/__init__.py", line 644, in find
    raise VersionConflict(dist, req)
pkg_resources.VersionConflict: (pip 20.1.1 (~/piptest/lib/python3.7/site-packages), Requirement.parse('pip==20.2b1'))

Actual: running pip results in an error
Expected: no error

Slightly longer repro steps:
https://gist.github.com/psybuzz/0c46c72e716d36ee3c492dec44680c19

From the VersionConflict error, I'm guessing that easy_install -U pip installed a newer version of pip (20.2b1) without removing the old one (20.1.1). Anything that requires 20.2.b1 then sees the old version actively distributed (20.1.1) and throws the error.

Since this happened recently, I'm wondering whether the recent changes on June 15 to easy_install (2b4b5de and 9bb1149) might have contributed to this issue.

Sorry if this isn't the right place to ask this question, but has anyone on the setuptools side seen this error?

[jaraco]

I suspect the offending behavior happened in this line of 9bb1149. In particular, removing the __requires__ line from the wrapper script means that the disambiguation of multiple versions is no longer triggered.

Although use of easy_install as an installer is discouraged, it has not yet been properly deprecated, so it's a little unfair to impose this incompatible change. It may also be the case that the presence of importlib_metadata or Python 3.8 would also be subject to this issue (or maybe not because it omits the version number when resolving the requirement).

I can get a fix out for this later this week (or sooner if it's urgent). What is the urgency of this issue? Are you able to manually work around it for now? Can you try pip install importlib_metadata and see if that help work around the issue (or if it remains)?

Have you considered not using easy_install? Why not use pip install -U pip (or python -m pip install -U pip for better compatibility on Windows)? That technique is better supported and should properly uninstall all versions.

[psybuzz]

Thanks for the timely response and explanation.

What is the urgency of this issue?

Non-urgent in the project I work on. This change motivated us to move migrate off easy_install and do python -m pip install -U pip as you suggested. Things are working again!

We have an automated script that ran the easy_install -U command, so it wasn't immediately clear what the culprit was. Seeing the warning in the logs was helpful though:
WARNING: The easy_install command is deprecated and will be removed in a future version.

I'm not familiar with Pypa's policies regarding deprecation, but my guess is that the urgency of this issue depends on how many users are using easy_install on a regular basis.

Can you try pip install importlib_metadata

Running pip ... at all will break, but if I do easy_install importlib_metadata, it fixes the issue, and commands like pip list work again.

[jaraco]

Things are working again!

Glad to hear it!

Running pip ... at all will break, but if I do easy_install importlib_metadata, it fixes the issue, and commands like pip list work again.

In that case, I recommend using python -m pip because that bypasses the easy_install wrappers and the underlying issue.

Given that the workaround is fairly straightforward and that as you revealed, easy-install is deprecated, I'm slightly inclined not to fix this issue.

On the other hand, it was an unintentional regression, so I'll probably just get a fix out so we can remove it later with intent and clarity.