[`pyproject.toml`: part 3] Add means to apply configuration from `pyproject.toml` by abravalheri · Pull Request #3067 · pypa/setuptools

abravalheri · 2022-02-01T12:34:08Z

This is part of #2970, split into several stacked PRs as requested in #2970 (review)
Follows #3066

This change targets adding a standardised project configuration format as initially proposed by PEP 621 (with the possibility of modification/evolution according to follow-up PEPs).

For the configuration fields not covered by the standards, the approach proposed in ini2toml is adopted.

This operation relies on vendorised external libraries tomli and validate-pyproject.

Summary of changes

Implement read_configuration from pyproject.toml - 3eb62ac
Expand dynamic entry_points from pyproject.toml - 112e8a9
Add means of applying config read from pyproject.toml to dist - 303791a
Add the apply_configuration API to setuptools.config.setupcfg - 43e3436
Test pyproject.toml config has the same effect as setup.cfg - c6907cb

Please see the commit message of each individual commit for more information about the changes.

Pull Request Checklist

Changes have tests
News fragment added in changelog.d/.
(See documentation for details)

webknjaz · 2022-02-09T13:45:55Z

@abravalheri FYI the PR title has a typo (missing "t" in "pyproject"), I can't fix it myself.

webknjaz · 2022-02-09T14:35:21Z

+
+@pytest.mark.parametrize("url", EXAMPLE_URLS)
+@pytest.mark.filterwarnings("ignore")
+def test_apply_pyproject_equivalent_to_setupcfg(url, monkeypatch, tmp_path):


Am I reading this right that this relies on network I/O? Maybe this could be marked as such?

Yes @webknjaz, this test will try to download a bunch of real world examples... It is possible to cache in the CI, but I haven't implemented that yet.

Is your suggestion adding a simple @pytest.mark.uses_network, or something more complex?

(when running the tests locally, the files will be cached)

webknjaz · 2022-02-09T14:40:12Z

+    return path
+
+
+def download(url, dest):


It's probably worth implementing a global cache and avoiding making fixture download part of testing. It should be possible to request cache download or --offline via a pytest CLI flag that could be implemented via a pytest hook callback.

Some of related ideas have been suggested in the past @ #2607

I think for the time being I will add a uses_network mark, so these tests can be skipped with -m "not uses_network" if necessary.

As a follow up, my intention is to add caching to the CI (locally the caching will already take place).

An --offline CLI flag would require a more comprehensive effort by checking from all existing tests which ones use the network, mark them with the uses_network mark and then doing a trick in conftest.py similar to the _skip_integration... So I would also leave it as "future work".

webknjaz · 2022-02-09T14:45:11Z

+    value = re.sub(r"^Project-URL: Homepage,.*$", "", value, flags=re.M)
+    # May be missing in original (relying on default) but backfilled in the TOML
+    value = re.sub(r"^Description-Content-Type:.*$", "", value, flags=re.M)
+    # ini2toml can automatically convert `tests_require` to `testing` extra


Why not "tests"? I would expect that "testing" contains helpers for the end-users to test their project using the given package while "tests" would correspond to its own internal test requirements.

I ended up going with testing because this seems to be popular in the community... I have the impression that this approach was influenced by the fact that tox docs explicitly mention testing.

For example setuptools and virtualenv use testing.

abravalheri · 2022-02-09T19:48:28Z

Thank you very much for the review @webknjaz.

Regarding the offline tests, I tried to mark the tests here so they can be skipped. I am afraid implementing --offline in pytest will be a much bigger scope and involve changes in existing tests, so I am deferring that for the future.

This change targets adding a standardised project configuration format as initially proposed by PEP 621 (with the possibility of modification/evolution according to follow-up PEPs). For the configuration fields not covered by the standards, the approach proposed in `ini2toml` is adopted. This operation relies on vendorised external libraries `tomli` and `validate-pyproject`.

This is the first step towards making setuptools understand `pyproject.toml` as a configuration file. The implementation deliberately allows splitting the act of loading the configuration from a file in 2 stages: the reading of the file itself and the expansion of directives (and other derived information).

The user might specify dynamic `entry-points` via a `file:` directive (a similar feature for `setup.cfg` is documented in [declarative config]). The changes introduced here add the ability to expand them when reading the configuration from `pyproject.toml`. [declarative config]: https://setuptools.pypa.io/en/latest/userguide/declarative_config.html

There is frequent an opinion in the community that `include_package_data=True` is a better default (and a quality of life improvement). Since we are migrating to a new configuration file, this change can be implemented in a backward compatible way (to avoid breaking existing packages): - Config from `setup.cfg` defaults to `include_package_data=False` - Config from `pyproject.toml` defaults to `include_package_data=True` This also takes advantage that `ini2toml` (the provided library for automatic conversion between `setup.cfg` and `pyproject.toml`) will backfill `include_package_data=False` when the field is missing.

Since the Distrubition and DistributionMetadata classes are modeled after (an old version of) core metadata, it is necessary to add a translation layer between them and the configuration read from pyproject.toml

The apply_configuration is implemented in a way that it is consistent for both pyproject.toml and setup.cfg

Co-authored-by: Henry Schreiner <HenrySchreinerIII@gmail.com>

Co-authored-by: Sviatoslav Sydorenko <wk.cvs.github@sydorenko.org.ua>

According to PEP 621, the backend should fill-in the content-type if the `readme` field is passed as a string. The value is derived from the extension of the file (an error should be raised when that is not possible). Previously all READMEs were wrongly assumed rst. This error was reported in: https://discuss.python.org/t/help-testing-experimental-features-in-setuptools/13821/4

webknjaz · 2024-12-06T01:04:14Z

@abravalheri so I rediscovered this file by accident and wanted to say that I saw a corner case with setup.cfg behaving differently from pyproject.toml in the case of multiple package roots (in that pyproject.toml works while setup.cfg doesn't): https://github.com/ansible/ansible/pull/81440/files#diff-fa602a8a75dc9dcc92261bac5f533c2a85e34fcceaff63b3a3a81d9acde2fc52R40-R42.

abravalheri requested a review from webknjaz February 1, 2022 12:34

abravalheri changed the title ~~[pyproject.toml: part 3] Add means to apply configuration from pyprojec.toml~~ [pyproject.toml: part 4] Add means to apply configuration from pyprojec.toml Feb 1, 2022

abravalheri mentioned this pull request Feb 1, 2022

[pyproject.toml: part 4] Integrate pyproject.toml configuration into existing classes #3068

Merged

2 tasks

abravalheri marked this pull request as ready for review February 3, 2022 10:22

abravalheri force-pushed the add-vendored-deps branch from 6f2d4fe to 9276edd Compare February 7, 2022 23:39

abravalheri force-pushed the dist-config-from-pyproject branch 2 times, most recently from 72fe662 to 5ef62fc Compare February 8, 2022 11:54

abravalheri changed the title ~~[pyproject.toml: part 4] Add means to apply configuration from pyprojec.toml~~ [pyproject.toml: part 4] Add means to apply configuration from pyproject.toml Feb 9, 2022

webknjaz reviewed Feb 9, 2022

View reviewed changes

Comment thread .gitignore Outdated

webknjaz reviewed Feb 9, 2022

View reviewed changes

Comment thread setuptools/tests/config/test_apply_pyprojecttoml.py Outdated

webknjaz reviewed Feb 9, 2022

View reviewed changes

Comment thread setuptools/tests/config/test_pyprojecttoml.py Outdated

webknjaz reviewed Feb 9, 2022

View reviewed changes

Comment thread setuptools/tests/config/test_pyprojecttoml.py Outdated

abravalheri force-pushed the add-vendored-deps branch from d1301c8 to e31b031 Compare February 9, 2022 19:38

abravalheri force-pushed the dist-config-from-pyproject branch from 5ef62fc to bfb2ff2 Compare February 9, 2022 19:44

abravalheri force-pushed the dist-config-from-pyproject branch 2 times, most recently from 9dcd88b to 5ecdb5b Compare February 10, 2022 20:18

abravalheri force-pushed the add-vendored-deps branch from 6cab284 to e440b9f Compare February 11, 2022 19:18

abravalheri force-pushed the dist-config-from-pyproject branch from 5ecdb5b to c2b80aa Compare February 11, 2022 19:30

abravalheri changed the title ~~[pyproject.toml: part 4] Add means to apply configuration from pyproject.toml~~ [pyproject.toml: part 3] Add means to apply configuration from pyproject.toml Feb 11, 2022

abravalheri mentioned this pull request Feb 11, 2022

Support pyproject.toml-style configuration (PEP 621) - Round 2 #2970

Merged

2 tasks

abravalheri force-pushed the add-vendored-deps branch from e440b9f to 93a3905 Compare February 18, 2022 22:40

abravalheri force-pushed the dist-config-from-pyproject branch from c2b80aa to 69ddad0 Compare February 18, 2022 22:40

abravalheri force-pushed the add-vendored-deps branch from 93a3905 to 4f50d08 Compare February 19, 2022 19:35

abravalheri force-pushed the dist-config-from-pyproject branch from 69ddad0 to 19b6ce2 Compare February 19, 2022 19:44

abravalheri and others added 17 commits March 5, 2022 14:30

Add means of applying config read from pyproject.toml to dist

9672a48

Since the Distrubition and DistributionMetadata classes are modeled after (an old version of) core metadata, it is necessary to add a translation layer between them and the configuration read from pyproject.toml

Add the apply_configuration API to setuptools.config.setupcfg

d7363d5

The apply_configuration is implemented in a way that it is consistent for both pyproject.toml and setup.cfg

Test pyproject.toml config has the same effect as setup.cfg

26a9264

Fix pyproject config when tool table is not present

051b825

Co-authored-by: Henry Schreiner <HenrySchreinerIII@gmail.com>

Remove no longer needed tomli import workaround

c927227

Update version of test dependency 'ini2toml'

905eed7

Prevent resource warnings in test_apply_pyprojecttoml

b426b2b

Co-authored-by: Sviatoslav Sydorenko <wk.cvs.github@sydorenko.org.ua>

Add a 'uses_network' marker to tests that require connectivity

e91969a

Update test dependency ini2toml to 0.8

9ee2697

Avoid failing due to 3rd party config in pyproject.toml

e5c5519

Add tests against "empty" pyproject.toml

5d4457e

Avoid using pkg_resources for entry points

cf32acb

Update test dependency ini2toml to 0.9

0497954

abravalheri force-pushed the add-vendored-deps branch from 4f50d08 to 7f68bb4 Compare March 5, 2022 14:40

abravalheri force-pushed the dist-config-from-pyproject branch from 979596a to 0497954 Compare March 5, 2022 14:41

abravalheri added this to the Add experimental support for project metadata via `pyproject.toml` milestone Mar 17, 2022

Base automatically changed from add-vendored-deps to support-pyproject-aggregate March 18, 2022 10:08

abravalheri merged commit 2cfcf0e into support-pyproject-aggregate Mar 18, 2022

abravalheri deleted the dist-config-from-pyproject branch March 18, 2022 10:09

stanislavlevin mentioned this pull request Apr 7, 2022

test_pyproject_support and setuptools 61.0.0 pypa/setuptools-scm#700

Closed

penguinpee mentioned this pull request Sep 30, 2022

Lower setuptools requirement (currently >=61) munin-monitoring/PyMunin3#4

Closed

webknjaz reviewed Dec 6, 2024

View reviewed changes

coderabbitai Bot mentioned this pull request Apr 14, 2026

ci: modernize workflows to OpenVoiceOS standards JarbasHiveMind/hivemind-redis-database#14

Merged

Uh oh!

Conversation

abravalheri commented Feb 1, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary of changes

Pull Request Checklist

Uh oh!

webknjaz commented Feb 9, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

abravalheri commented Feb 9, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

abravalheri commented Feb 1, 2022 •

edited

Loading