Debian-specific behavior lost when Setuptools adopts distutils

[prometheanfire]

pip install -e $PACKAGE_PATH installs commands to /usr/bin and not /usr/local/bin on Ubuntu as it did in the past.

pip install $PACKAGE_PATH continues to install to /usr/local/bin as expected.

Openstack downstream has temporarily capped setuptools until we know if this is expected behaviour or not. Please see http://lists.openstack.org/pipermail/openstack-discuss/2020-July/015779.html for more context.

[jaraco]

Thanks for the report. I'm initially unable to replicate the reported baseline behavior:

$ docker run -it jaraco/multipy-tox bash -c 'git clone -q https://github.com/pypa/twine; python3 -m pip-run -q "setuptools<48" -- -m pip install -q --no-deps -e twine; ls /usr/bin/twine /usr/local/bin/twine'
ls: cannot access '/usr/local/bin/twine': No such file or directory
/usr/bin/twine

note: multipy-tox is Ubuntu Focal with Python 3.8.2 installed

In that command, I use Docker to install setuptools<48 to then run pip install -e twine (a local checkout), which runs yet no command exist as /usr/local/bin/twine, but the command does exist as /usr/bin/twine.

So it seems maybe the discriminating factor is not the Setuptools 48 release (though its change is plausibly implicated). In Setuptools 49.1, I've (temporarily) disabled the functionality introduced in Setuptools 48. Does that correct the issue (in your environment)?

What other factors could be causing this unexpected behavior, even with older Setuptools? How can I replicate the issue?

[cboylan]

This shows the behavior:

sudo docker run -it --rm ubuntu:20.04 bash -c 'apt-get update && apt-get install -y python3-pip git && pip3 install -U pip && pip --version && pip install -U "setuptools<49" && git clone https://github.com/pypa/twine && pip install ./twine && which twine && pip uninstall -y twine && pip install -e ./twine && which twine'

Using setuptools 49.1.0 seems to fix it (the difference between the commands is the version of setuptools installed):

sudo docker run -it --rm ubuntu:20.04 bash -c 'apt-get update && apt-get install -y python3-pip git && pip3 install -U pip && pip --version && pip install -U "setuptools==49.1.0" && git clone https://github.com/pypa/twine && pip install ./twine && which twine && pip uninstall -y twine && pip install -e ./twine && which twine'

I won't bother to paste the output of the commands here because it is quite verbose, but if that would be useful I can add it as attachments I guess.

[jaraco]

I see the mistake with my example above; I was using pip-run to temporarily override the version of setuptools, but twine uses PEP 517 to build, so the build isolation causes that value to be discarded. If I'm using pip-run to override build requirements, I should also pass --no-build-isolation to pip install.

I'm able to quickly replicate the issue with this command:

$ docker run -it jaraco/multipy-tox bash -c 'git clone -q https://github.com/pypa/twine; python3 -m pip install -q "setuptools==48"; python3 -m pip install -q --no-deps -e twine; which twine'
/usr/bin/twine

[jaraco]

With this command, I'm able to replicate the behavior natively in setuptools:

$ docker run -it jaraco/multipy-tox bash -c 'git clone -q https://github.com/pypa/twine; cd twine; python3 -m pip install -q "setuptools==48"; python3 setup.py develop --no-deps; which twine'
running develop
running egg_info
creating twine.egg-info
writing twine.egg-info/PKG-INFO
writing dependency_links to twine.egg-info/dependency_links.txt
writing entry points to twine.egg-info/entry_points.txt
writing requirements to twine.egg-info/requires.txt
writing top-level names to twine.egg-info/top_level.txt
writing manifest file 'twine.egg-info/SOURCES.txt'
writing manifest file 'twine.egg-info/SOURCES.txt'
running build_ext
Creating /usr/lib/python3.8/site-packages/twine.egg-link (link to .)
Adding twine 3.2.1.dev1+g141428b to easy-install.pth file
Installing twine script to /usr/bin

Installed /twine
/usr/bin/twine

[jaraco]

It seems that Debian and Ubuntu keep their own forks of distutils. I copied /usr/lib/python3.8/distutils over setuptools/_distutils and found diffs like this:

diff --git a/setuptools/_distutils/command/install.py b/setuptools/_distutils/command/install.py
index 13feeb89..2934862f 100644
--- a/setuptools/_distutils/command/install.py
+++ b/setuptools/_distutils/command/install.py
@@ -30,33 +30,33 @@ WINDOWS_SCHEME = {
 INSTALL_SCHEMES = {
     'unix_prefix': {
         'purelib': '$base/lib/python$py_version_short/site-packages',
-        'platlib': '$platbase/$platlibdir/python$py_version_short/site-packages',
+        'platlib': '$platbase/lib/python$py_version_short/site-packages',
         'headers': '$base/include/python$py_version_short$abiflags/$dist_name',
         'scripts': '$base/bin',
         'data'   : '$base',
         },
+    'unix_local': {
+        'purelib': '$base/local/lib/python$py_version_short/dist-packages',
+        'platlib': '$platbase/local/lib/python$py_version_short/dist-packages',
+        'headers': '$base/local/include/python$py_version_short/$dist_name',
+        'scripts': '$base/local/bin',
+        'data'   : '$base/local',
+        },
+    'deb_system': {
+        'purelib': '$base/lib/python3/dist-packages',
+        'platlib': '$platbase/lib/python3/dist-packages',
+        'headers': '$base/include/python$py_version_short/$dist_name',
+        'scripts': '$base/bin',
+        'data'   : '$base',
+        },
     'unix_home': {
         'purelib': '$base/lib/python',
-        'platlib': '$base/$platlibdir/python',
+        'platlib': '$base/lib/python',
         'headers': '$base/include/python/$dist_name',
         'scripts': '$base/bin',
         'data'   : '$base',
         },
     'nt': WINDOWS_SCHEME,
-    'pypy': {
-        'purelib': '$base/site-packages',
-        'platlib': '$base/site-packages',
-        'headers': '$base/include/$dist_name',
-        'scripts': '$base/bin',
-        'data'   : '$base',
-        },
-    'pypy_nt': {
-        'purelib': '$base/site-packages',
-        'platlib': '$base/site-packages',
-        'headers': '$base/include/$dist_name',
-        'scripts': '$base/Scripts',
-        'data'   : '$base',
-        },
     }
 
 # user site schemes
@@ -145,6 +145,9 @@ class install(Command):
 
         ('record=', None,
          "filename in which to record list of installed files"),
+
+        ('install-layout=', None,
+         "installation layout to choose (known values: deb, unix)"),
         ]
 
     boolean_options = ['compile', 'force', 'skip-build']
@@ -165,6 +168,7 @@ class install(Command):
         self.exec_prefix = None
         self.home = None
         self.user = 0
+        self.prefix_option = None
 
         # These select only the installation base; it's up to the user to
         # specify the installation scheme (currently, that means supplying
@@ -186,6 +190,10 @@ class install(Command):
         self.install_userbase = USER_BASE
         self.install_usersite = USER_SITE
 
+        # enable custom installation, known values: deb
+        self.install_layout = None
+        self.multiarch = None
+        
         self.compile = None
         self.optimize = None
 
@@ -312,7 +320,6 @@ class install(Command):
                             'sys_exec_prefix': exec_prefix,
                             'exec_prefix': exec_prefix,
                             'abiflags': abiflags,
-                            'platlibdir': getattr(sys, 'platlibdir', 'lib'),
                            }
 
         if HAS_USER_SITE:
@@ -428,6 +435,7 @@ class install(Command):
             self.install_base = self.install_platbase = self.home
             self.select_scheme("unix_home")
         else:
+            self.prefix_option = self.prefix
             if self.prefix is None:
                 if self.exec_prefix is not None:
                     raise DistutilsOptionError(
@@ -442,7 +450,28 @@ class install(Command):
 
             self.install_base = self.prefix
             self.install_platbase = self.exec_prefix
-            self.select_scheme("unix_prefix")
+            if self.install_layout:
+                if self.install_layout.lower() in ['deb']:
+                    import sysconfig
+                    self.multiarch = sysconfig.get_config_var('MULTIARCH')
+                    self.select_scheme("deb_system")
+                elif self.install_layout.lower() in ['unix']:
+                    self.select_scheme("unix_prefix")
+                else:
+                    raise DistutilsOptionError(
+                        "unknown value for --install-layout")
+            elif ((self.prefix_option and
+                   os.path.normpath(self.prefix) != '/usr/local')
+                  or sys.base_prefix != sys.prefix
+                  or 'PYTHONUSERBASE' in os.environ
+                  or 'VIRTUAL_ENV' in os.environ
+                  or 'real_prefix' in sys.__dict__):
+                self.select_scheme("unix_prefix")
+            else:
+                if os.path.normpath(self.prefix) == '/usr/local':
+                    self.prefix = self.exec_prefix = '/usr'
+                    self.install_base = self.install_platbase = '/usr'
+                self.select_scheme("unix_local")
 
     def finalize_other(self):
         """Finalizes options for non-posix platforms"""
@@ -469,12 +498,6 @@ class install(Command):
     def select_scheme(self, name):
         """Sets the install directories by applying the install schemes."""
         # it's the caller's problem if they supply a bad name!
-        if (hasattr(sys, 'pypy_version_info') and
-                not name.endswith(('_user', '_home'))):
-            if os.name == 'nt':
-                name = 'pypy_nt'
-            else:
-                name = 'pypy'
         scheme = INSTALL_SCHEMES[name]
         for key in SCHEME_KEYS:
             attrname = 'install_' + key

So when Setuptools adopts distutils from the standard library and uses it, any tweaks that downstream packagers have made to distutils will be lost.

Where are these tweaks managed? Can I get some help from Debian and/or Ubuntu to apply these tweaks to distutils?

[pradyunsg]

Can I get some help from Debian and/or Ubuntu to apply these tweaks to distutils?

/cc @kitterma (one of the Debian Python folks)

[pradyunsg]

Also relevant: https://sources.debian.org/patches/python3.8/

Specifically: https://sources.debian.org/patches/python3.8/3.8.4%7Erc1-1/distutils-install-layout.diff/

[jaraco]

I took a quick look at the patch and it looks pretty manageable. However, the options seem to change the default behavior. What we would need for distutils is something that behaves the same as currently by default, but on Debian honors the alternate behaviors.

[doko42]

Making the installation into /usr the default is definitely not appreciated. I think we discussed this at length at one of the Cleveland sprints.

The behavior that you don't want to have is to install, remove, or update a package into /usr which was installed using the Debian package manager (apt/dpkg). Afaics this is now exactly the thing which happens with the current setuptools behavior. It's unfortunate that a change like this is again introduced. I don't think that the solution is to change any default for setuptools, but to make pip aware of dpkg/apt managed installations.

[jaraco]

Here, Setuptools is only relying on the distutils behavior from CPython. All that's needed is to port the Debian-specific behavior into pypa/distutils and (especially if there's a test capturing the expectation), Setuptools will ever-after support that behavior without the need for Debian to patch the software when installing it. If the Debian behavior is preferable for more environments, the project can consider employing it there as well.

To be certain, this change was not expected and was disabled within 24 hours. Now all we're seeking is to settle on a long-term solution in the pypa/distutils repo that can become the reference implementation without any downstream patches.

[jaraco]

I think we discussed this at length at one of the Cleveland sprints.

By the way, I remember the discussions and enjoyed our exchange. Unfortunately, so much time has passed that I haven't retained any actionable points from our discussion.

Do you have a recommendation? I don't think the diverse community of Setuptools users can readily adopt the Debian patch wholesale. Perhaps Setuptools/distutils could adopt those changes over time, giving other operating systems time to adapt to the approach Debian uses... or maybe those platforms have sound reasons for not having those changes. In any case, I believe the near-term solution, to unblock Setuptools from adopting distutils, would be for pypa/distutils to adopt the Debian patch but enable the Debian default settings only on Debian. Can you advise how to reliably detect hosts that would opt into this behavior?