[FR] Improve error feedback for improper one-liner requirement with environment markers in `setup.cfg`

[frenzymadness]

setuptools version

63.2.0

Python version

3.9 and 3.11

OS

Fedora Linux

Additional environment information

No response

Description

Project: https://github.com/thoth-station/micropipenv
The line I have a problem with: https://github.com/thoth-station/micropipenv/blob/2127293c4c9fa344fd1535c5189c39ca7ba4594d/setup.cfg#L40

It seems to me that there is a difference between this:

[options.extras_require]
toml = toml;python_version<"3.11"

and this:

[options.extras_require]
toml =
    toml;python_version<"3.11"

Micropipenv version 1.4.0 contains the [toml] definition on a single line and when you try to install micropipenv[toml], it does not work (Python 3.9.13):

Collecting micropipenv[toml]==1.4.0
  Using cached micropipenv-1.4.0-py3-none-any.whl (38 kB)
Requirement already satisfied: pip>=9 in /home/lbalhar/.virtualenvs/micropipenv/lib/python3.9/site-packages (from micropipenv[toml]==1.4.0) (22.2)
Collecting toml
  Using cached toml-0.10.2-py2.py3-none-any.whl (16 kB)
ERROR: Could not find a version that satisfies the requirement python-version<"3.11"; extra == "toml" (from micropipenv[toml]) (from versions: 0.0.2)
ERROR: No matching distribution found for python-version<"3.11"; extra == "toml"

the same happens when you use Python 3.11.

But when I use the multiline definition:

[options.extras_require]
toml =
    toml;python_version<"3.11"

it works correctly and omits toml in env with Python 3.11 and installs it if Python<3.11.

When I build a sdist from the package, there is a difference in requires.txt in egg-info directory:
single line definition:

pip>=9

[toml]
toml
python_version<"3.11"

two lines:

pip>=9

[toml]

[toml:python_version < "3.11"]
toml

Expected behavior

I'd expect the two definitions mentioned above to behave the same way.

How to Reproduce

You can try to install "micropipenv[toml]==1.4.0" and then you can change the extras_require definition as described above to see that it fixes the problem.

Output

$ tail -n 3 setup.cfg

[options.extras_require]
toml = toml;python_version<"3.11"

$ pip install .[toml]
Processing /home/lbalhar/Software/micropipenv
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: pip>=9 in /home/lbalhar/.virtualenvs/micropipenv/lib/python3.9/site-packages (from micropipenv==1.4.0) (22.2)
Collecting toml
  Using cached toml-0.10.2-py2.py3-none-any.whl (16 kB)
ERROR: Could not find a version that satisfies the requirement python-version<"3.11"; extra == "toml" (from micropipenv[toml]) (from versions: 0.0.2)
ERROR: No matching distribution found for python-version<"3.11"; extra == "toml"

## changed the definition

$ tail -n 3 setup.cfg
[options.extras_require]
toml =
    toml;python_version<"3.11"

$ pip install .[toml]
Processing /home/lbalhar/Software/micropipenv
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: pip>=9 in /home/lbalhar/.virtualenvs/micropipenv/lib/python3.9/site-packages (from micropipenv==1.4.0) (22.2)
Collecting toml
  Using cached toml-0.10.2-py2.py3-none-any.whl (16 kB)
Building wheels for collected packages: micropipenv
  Building wheel for micropipenv (pyproject.toml) ... done
  Created wheel for micropipenv: filename=micropipenv-1.4.0-py3-none-any.whl size=23303 sha256=b645cc71f7404f367dabe8c203cf80025f3080c7d2ac6a929aec3ae5bbab2a6a
  Stored in directory: /tmp/pip-ephem-wheel-cache-zh2n98et/wheels/2a/c1/ee/2caed205bb1a97b553aa44d9a5f1966c5bec9edea862cc2c72
Successfully built micropipenv
Installing collected packages: toml, micropipenv
Successfully installed micropipenv-1.4.0 toml-0.10.2

[frenzymadness]

From the configparser perspective, the difference is this:

>>> c['options.extras_require']["toml"]
'toml;python_version<"3.11"'

versus

>>> c['options.extras_require']["toml"]
'\ntoml;python_version<"3.11"'

[hroncok]

The (broken) wheel has this in the metadata:

Requires-Dist: toml ; extra == 'toml'
Requires-Dist: python-version (<"3.11") ; extra == 'toml'

[abravalheri]

Hi @frenzymadness , this is a limitation in terms of setup.cfg that is explained in the docs:

In the extras_require section, values are parsed as list-semi. This implies that in order to include markers, they must be dangling

The rationale is that if you don't include newlines in the value, the character used to split different requirements is ;.

[frenzymadness]

I've fixed the problem in release 1.4.1 so now it's even easier to reproduce. The old version fails to install no matter which version of Python you use:
Python 3.11:

$ pip install "micropipenv[toml]==1.4.0"
Collecting micropipenv[toml]==1.4.0
  Using cached micropipenv-1.4.0-py3-none-any.whl (38 kB)
Requirement already satisfied: pip>=9 in ./venv/lib/python3.11/site-packages (from micropipenv[toml]==1.4.0) (21.3.1)
ERROR: Could not find a version that satisfies the requirement python-version<"3.11"; extra == "toml" (from micropipenv[toml]) (from versions: 0.0.2)
ERROR: No matching distribution found for python-version<"3.11"; extra == "toml"

Python 3.9

$ pip install "micropipenv[toml]==1.4.0"
Collecting micropipenv[toml]==1.4.0
  Using cached micropipenv-1.4.0-py3-none-any.whl (38 kB)
Requirement already satisfied: pip>=9 in ./venv/lib/python3.9/site-packages (from micropipenv[toml]==1.4.0) (21.3.1)
Collecting toml
  Using cached toml-0.10.2-py2.py3-none-any.whl (16 kB)
ERROR: Could not find a version that satisfies the requirement python-version<"3.11"; extra == "toml" (from micropipenv[toml]) (from versions: 0.0.2)
ERROR: No matching distribution found for python-version<"3.11"; extra == "toml"

The new version works correctly and omits toml for Python 3.11:

$ pip install "micropipenv[toml]==1.4.1"
Collecting micropipenv[toml]==1.4.1
  Downloading micropipenv-1.4.1-py3-none-any.whl (38 kB)
Requirement already satisfied: pip>=9 in ./venv/lib/python3.11/site-packages (from micropipenv[toml]==1.4.1) (21.3.1)
Installing collected packages: micropipenv
Successfully installed micropipenv-1.4.1

Python 3.9

$ pip install "micropipenv[toml]==1.4.1"
Collecting micropipenv[toml]==1.4.1
  Using cached micropipenv-1.4.1-py3-none-any.whl (38 kB)
Requirement already satisfied: pip>=9 in ./venv/lib/python3.9/site-packages (from micropipenv[toml]==1.4.1) (21.3.1)
Collecting toml
  Using cached toml-0.10.2-py2.py3-none-any.whl (16 kB)
Installing collected packages: toml, micropipenv
Successfully installed micropipenv-1.4.1 toml-0.10.2

[hroncok]

My assumption is that:

toml = toml;python_version<"3.11"

Is read as:

toml = <requirement1>;<requirement2>

While:

toml =
    toml;python_version<"3.11"

Is read as:

toml =
    <requirement1>

This behavior seems consistent with the examples from the docs:

[options.extras_require]
pdf = ReportLab>=1.2; RXP
rest = docutils>=0.3; pack ==1.1, ==1.3

Albeit, quite confusing with environment markers.

[hroncok]

Hi @frenzymadness , this is a limitation in terms of setup.cfg that is explained in the docs:

In the extras_require section, values are parsed as list-semi. This implies that in order to include markers, they must be dangling

The rationale is that if you don't include newlines in the value, the character used to split different requirements is ;.

Oh, I sent my comment before seeing this. I agree that this is a known limitation, however I would argue that it is a trap.

We've discussed with @frenzymadness and we belive a warning might be presented when the entire line:

• contains exactly 1 ;
• parses as a valid requirement string
• the part after ; parses as a valid environment marker

WDYT?

[frenzymadness]

Thanks for the info @abravalheri ! What do you think about an automated check for this? I think it'd be easy to implement it. Something like: If you have extras_require on a single line and after split by ; one of the requirements contains env marker, issue a warning (or error?).

[abravalheri]

@hroncok yes, that is precisely the documented behaviour.

extras_require is parsed as a list-semi value, which means that if the value does not contain any new line character, the value will be split using the ; separator.

@frenzymadness , I am afraid this behaviour is not a bug but rather an intentional design decision.

If you prefer having more clarity in terms of the configuration language syntax, maybe you would like to try pyproject.toml¹. Just please be aware that due to PEP 621, this kind of configuration is handled in a much more strict way.

Footnotes

1. Differently from INI files, the TOML language has first class support for lists, so setuptools don't need to perform any arbitrary string split operation. I think this is one of the biggest "pros" of using TOML (although the language also have it's own perks). ↩

[abravalheri]

a warning might be presented ... the part after ; parses as a valid environment marker

That could be a nice addition. Thank you very much @hroncok and @frenzymadness . You one of you guys would like to give it a try in a PR?

[frenzymadness]

We will try it soon.

[frenzymadness]

Before I try to create a test for it, would something like this be considered as an acceptable solution?

--- a/setuptools/config/setupcfg.py
+++ b/setuptools/config/setupcfg.py
@@ -10,11 +10,13 @@ import functools
 from collections import defaultdict
 from functools import partial
 from functools import wraps
+from itertools import chain
 from typing import (TYPE_CHECKING, Callable, Any, Dict, Generic, Iterable, List,
                     Optional, Tuple, TypeVar, Union)
 
 from distutils.errors import DistutilsOptionError, DistutilsFileError
 from setuptools.extern.packaging.version import Version, InvalidVersion
+from setuptools.extern.packaging.markers import Marker, InvalidMarker
 from setuptools.extern.packaging.specifiers import SpecifierSet
 from setuptools._deprecation_warning import SetuptoolsDeprecationWarning
 
@@ -702,6 +704,19 @@ class ConfigOptionsHandler(ConfigHandler["Distribution"]):
             section_options,
             self._parse_requirements_list,
         )
+
+        for requirement in chain(*parsed.values()):
+            try:
+                Marker(requirement)
+            except InvalidMarker:
+                pass
+            else:
+                msg = ( "One of the parsed requirements in extras_require section "
+                       f"looks like a valid environment marker: '{requirement}'\n"
+                        "Make sure that the config is correct and check "
+                        "https://setuptools.pypa.io/en/latest/userguide/declarative_config.html#opt-2")
+                warnings.warn(msg, UserWarning)
+
         self['extras_require'] = parsed
 
     def parse_section_data_files(self, section_options):

implemented in parse_section_extras_require.

It basically checks all the parsed requirements – it actually tries to create a marker from them – and reports a warning if it succeedes.