Version 1.2 breaks pypa/build on Python 3.8 and 3.9 #206

henryiii · 2024-10-06T04:27:21Z

1.2 has broken build's test suite when using backend-path on Python 3.8 and 3.9 (and check-sdist's setuptools test, though I just dropped the setuptools test there). See pypa/build#820, python/importlib_metadata#508, henryiii/check-sdist#55. I've finally narrowed it down to this package's 1.2 release.

Backend path is ".", and setuptools is making test_no_prepare.egg-info, which is then getting picked up as a nameless distribution, which causes a confusing error when trying to get the name and falling back on None, then not handling the None case.

Originally posted by @henryiii in #199 (comment)

Traceback (most recent call last):
  File "/build/.tox/py39/lib/python3.9/site-packages/importlib_metadata/compat/py39.py", line 19, in normalized_name
    return dist._normalized_name
AttributeError: 'PathDistribution' object has no attribute '_normalized_name'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/build/.tox/py39/lib/python3.9/site-packages/pyproject_hooks/_in_process/_in_process.py", line 389, in <module>
    main()
  File "/build/.tox/py39/lib/python3.9/site-packages/pyproject_hooks/_in_process/_in_process.py", line 373, in main
    json_out["return_val"] = hook(**hook_input["kwargs"])
  File "/build/.tox/py39/lib/python3.9/site-packages/pyproject_hooks/_in_process/_in_process.py", line 280, in build_wheel
    return _build_backend().build_wheel(
  File "/build/.tox/py39/lib/python3.9/site-packages/setuptools/build_meta.py", line 421, in build_wheel
    return self._build_with_temp_dir(
  File "/build/.tox/py39/lib/python3.9/site-packages/setuptools/build_meta.py", line 403, in _build_with_temp_dir
    self.run_setup()
  File "/build/.tox/py39/lib/python3.9/site-packages/setuptools/build_meta.py", line 318, in run_setup
    exec(code, locals())
  File "<string>", line 1, in <module>
  File "/build/.tox/py39/lib/python3.9/site-packages/setuptools/__init__.py", line 117, in setup
    return distutils.core.setup(**attrs)
  File "/build/.tox/py39/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 183, in setup
    return run_commands(dist)
  File "/build/.tox/py39/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 199, in run_commands
    dist.run_commands()
  File "/build/.tox/py39/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 954, in run_commands
    self.run_command(cmd)
  File "/build/.tox/py39/lib/python3.9/site-packages/setuptools/dist.py", line 950, in run_command
    super().run_command(command)
  File "/build/.tox/py39/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 973, in run_command
    cmd_obj.run()
  File "/build/.tox/py39/lib/python3.9/site-packages/setuptools/command/bdist_wheel.py", line 433, in run
    self.run_command("install")
  File "/build/.tox/py39/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 316, in run_command
    self.distribution.run_command(command)
  File "/build/.tox/py39/lib/python3.9/site-packages/setuptools/dist.py", line 950, in run_command
    super().run_command(command)
  File "/build/.tox/py39/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 973, in run_command
    cmd_obj.run()
  File "/build/.tox/py39/lib/python3.9/site-packages/setuptools/command/install.py", line 91, in run
    return super().run()
  File "/build/.tox/py39/lib/python3.9/site-packages/setuptools/_distutils/command/install.py", line 706, in run
    self.run_command(cmd_name)
  File "/build/.tox/py39/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 316, in run_command
    self.distribution.run_command(command)
  File "/build/.tox/py39/lib/python3.9/site-packages/setuptools/dist.py", line 950, in run_command
    super().run_command(command)
  File "/build/.tox/py39/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 973, in run_command
    cmd_obj.run()
  File "/build/.tox/py39/lib/python3.9/site-packages/setuptools/command/install_egg_info.py", line 32, in run
    self.run_command('egg_info')
  File "/build/.tox/py39/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 316, in run_command
    self.distribution.run_command(command)
  File "/build/.tox/py39/lib/python3.9/site-packages/setuptools/dist.py", line 950, in run_command
    super().run_command(command)
  File "/build/.tox/py39/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 973, in run_command
    cmd_obj.run()
  File "/build/.tox/py39/lib/python3.9/site-packages/setuptools/command/egg_info.py", line 302, in run
    for ep in metadata.entry_points(group='egg_info.writers'):
  File "/build/.tox/py39/lib/python3.9/site-packages/importlib_metadata/__init__.py", line 1051, in entry_points
    return EntryPoints(eps).select(**params)
  File "/build/.tox/py39/lib/python3.9/site-packages/importlib_metadata/__init__.py", line 1048, in <genexpr>
    eps = itertools.chain.from_iterable(
  File "/build/.tox/py39/lib/python3.9/site-packages/importlib_metadata/_itertools.py", line 17, in unique_everseen
    k = key(element)
  File "/build/.tox/py39/lib/python3.9/site-packages/importlib_metadata/compat/py39.py", line 23, in normalized_name
    return Prepared.normalize(getattr(dist, "name", None) or dist.metadata['Name'])
  File "/build/.tox/py39/lib/python3.9/site-packages/importlib_metadata/__init__.py", line 889, in normalize
    return re.sub(r"[-_.]+", "-", name).lower().replace('-', '_')
  File "/usr/local/lib/python3.9/re.py", line 210, in sub
    return _compile(pattern, flags).sub(repl, string, count)
TypeError: expected string or bytes-like object

The text was updated successfully, but these errors were encountered:

layday · 2024-10-06T07:37:51Z

It would appear #199 returns a stdlib (importlib.metadata) distribution which setuptools then attempts to operate on using the backport (importlib_metadata) and the two are incompatible on Python 3.8 and .9.

pradyunsg · 2024-10-06T09:19:07Z

/cc @jaraco @takluyver @abravalheri for awareness.

pradyunsg · 2024-10-06T10:30:04Z

Responding to #199 (comment):

I'm sorely tempted to declare that pyproject-hooks now requires Python 3.10 or above, and you should just pin whichever version works for you on older Python versions. I'm normally pretty keen on backwards compatibility, so I'd welcome a better solution, but I don't have the time or energy to wade through importlib stuff to figure this out. And I don't know how we can trust that the next solution won't just lead to another of these bugs popping up somewhere.

I have two concerns with that:

pip won't be able to upgrade to a newer version of pyproject_hooks, because pip can't be dropping support for Python 3.9 just yet. So, the underlying pip fails to build project from backend-path when MetaPathFinder is present and installing from VCS url pip#11812 issue can't be resolved until at least an year from now. Plus, pip won't be able to pull in any bugfixes etc from this project outside of maintaining its own fork... which is unappealing to say the least. 😓
If we do this, we should yank 1.2.0 and release a 1.3.0 with the right python-requires, so that anyone using 3.8/3.9 isn't going to get a broken release by default.

pfmoore · 2024-10-06T10:46:14Z

My inclination is to say that this is either a bug in setuptools (which should spot when it gets a stdlib object and deal with it) or in the importlib_metadata backport (which should offer some sort of compatibility).

None of which helps that much, unless those projects can release fixes ASAP, but at the same time, I sympathise with @takluyver's point that if we can't rely on being able to safely pass distribution objects around, we're going to be at risk of yet more areas of incompatibility popping up.

takluyver · 2024-10-06T11:31:08Z

If we do this, we should yank 1.2.0 and release a 1.3.0 with the right python-requires...

We might then need to yank 1.1.0 as well, since that was already reported as a regression (#192) and 1.2.0 was meant to fix it. And then I presume people would be upset about the problem that #165 was written to fix again. I totally agree that I don't want to do this.

A slightly less drastic version might be to change the conditional added in #199 - if sys.version_info >= (3, 8): - to (3, 10). This would mean no importlib[._]metadata integration on Python 3.8 or 3.9 (like v1.1), but keep the integration (like v1.2) for newer Pythons.

If setuptools is committed to using the backport on 3.8 & 3.9, we could decide that compatibility with setuptools is more important than anything else and insist on using the backport for those versions as well. But that presumably breaks other code that does the right thing and uses the stdlib version. And it's an extra headache for anyone who cares about bootstrapping, since importlib_metadata needs setuptools & setuptools_scm to build it. I really don't like that option.

layday · 2024-10-06T11:41:57Z

Has this issue actually been observed anywhere outside of a contrived test case? Build re-exposes setuptools without the prepare hook to test reading a project’s metadata from a wheel. Unless this incompatibility is causing builds to fail in the wild, yanking’s probably an overreaction at this point. Sent with [Proton Mail](https://proton.me/mail/home) secure email.

…

On Sunday, 6 October 2024 at 14:31, Thomas Kluyver ***@***.***> wrote: > If we do this, we should yank 1.2.0 and release a 1.3.0 with the right python-requires... We might then need to yank 1.1.0 as well, since that was already reported as a regression ([#192](#192)) and 1.2.0 was meant to fix it. And then I presume people would be upset about the problem that [#165](#165) was written to fix again. I totally agree that I don't want to do this. A slightly less drastic version might be to change the conditional added in [#199](#199) - if sys.version_info >= (3, 8): - to (3, 10). This would mean no importlib[._]metadata integration on Python 3.8 or 3.9 (like v1.1), but keep the integration (like v1.2) for newer Pythons. If setuptools is committed to using the backport on 3.8 & 3.9, we could decide that compatibility with setuptools is more important than anything else and insist on using the backport for those versions as well. But that presumably breaks other code that does the right thing and uses the stdlib version. And it's an extra headache for anyone who cares about bootstrapping, since importlib_metadata needs setuptools & setuptools_scm to build it. I really don't like that option. — Reply to this email directly, [view it on GitHub](#206 (comment)), or [unsubscribe](https:/notifications/unsubscribe-auth/AHNRFWFPYWAJWNRDMA2U44DZ2ENRFAVCNFSM6AAAAABPN6QQT2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGOJVGQYDGMRUGY). You are receiving this because you commented.Message ID: ***@***.***>

henryiii · 2024-10-06T14:24:04Z

It broke check-sdist’s tests too, where it was just building an SDist for setuptools. I believe it breaks any package that has a local backend at “.” and uses setuptools. Including setuptools itself (https:/pypa/setuptools/blob/66a8aee91de7ed8b0f819fb836e5543212e4b860/pyproject.toml#L4) and setuptools-scm. I’ve verified locally that you can’t build setuptools-scm anymore. Setuptools-scm uses uv to build SDists & wheels now, which is why their CI hasn't broken this last week.

You can find others at https:/search?type=code&q=path%3Apyproject.toml+backend-path - look for the ones with setuptools and “.”.

nanonyme · 2024-10-06T15:00:16Z

Oh. It fixed setuptools local backend at "." based on our actual usage of build to bootstrap setuptools at least on newer Python. Wait, is this only about sdist? Because we don't build any sdists through wheel in our deptree since the only point of building sdist is redistributing sources. We only build wheels.

henryiii · 2024-10-06T15:18:52Z

This is only about Python 3.8 and 3.9, everything works fine after that.

It breaks both:

$ git clone https:/cvxpy/cvxpy
$ cd cvxpy
$ pipx run --python 3.9 build --sdist
...
TypeError: expected string or bytes-like object
$ pipx run --python 3.9 build --wheel
...
TypeError: expected string or bytes-like object
$ pipx run --python 3.10 build --sdist
...
Successfully built cvxpy-1.6.0.tar.gz

If you avoid the new pyproject-hooks, it works:

$ uvx --python 3.9 --from build --with pyproject-hooks==1.1.0 pyproject-build --sdist
...
Successfully built cvxpy-1.6.0.tar.gz

You have to remove the *.dist-info dir after a successful build, however, otherwise it starts working.

jaraco · 2024-10-06T20:29:46Z

It would appear #199 returns a stdlib (importlib.metadata) distribution which setuptools then attempts to operate on using the backport (importlib_metadata) and the two are incompatible on Python 3.8 and .9.

As a reminder, importlib.metadata prior to Python 3.10 was considered "provisional", so it's probably preferable to use importlib_metadata at least on those versions.

importlib_metadata has attempted to accommodate objects from those implementations, but it's complicated due to the two API surfaces that library manages (providers and consumers). In python/importlib_metadata#486, we're exploring ways to shore up some of those relationships, but that still won't fix the fact that importlib.metadata on Python 3.9 and earlier will never honor the newer interfaces and guarantees of later releases. Later versions of importlib_metadata will provide an error earlier when the Name key is missing, but only for Distributions constructed from importlib_metadata.

That said, an egg-info directory containing no metadata is in fact invalid, right? Shouldn't the solution be to address the root cause (prevent that invalid metadata from existing)? What is the origin of these empty metadata directories?

jaraco · 2024-10-06T20:37:04Z

Oh, do we understand why this issue only affects Python 3.9 and earlier? If the underlying problem is a distribution with no Name, I'd expect that also to cause problems. I guess it doesn't because Distribution.name (or ._normalized_name) happens to return None for these invalid distributions and because the unique check when scanning entry points happens not to care if it gets None where it expects a str. That means that this issue will also affect Python 3.14, where importlib.metadata.Distribution.name will now raise a KeyError when no name is present in the metadata.

abravalheri · 2024-10-06T22:26:52Z

Isn't this the same problem we were discussing in #195 (comment)?

We are in the "complicated zone" when importlib_metadata exists in path.

(Frustratingly, I tested the patch against the setuptools test suite and it passed at that point in time https:/abravalheri/setuptools/pull/14/files)

abravalheri · 2024-10-07T10:09:48Z

That said, an egg-info directory containing no metadata is in fact invalid, right? Shouldn't the solution be to address the root cause (prevent that invalid metadata from existing)? What is the origin of these empty metadata directories?

@jaraco, I suspect that this parish aspect is an issue in setuptools because it uses entry-points to generate the .egg-info directory itself. So setuptools create an empty .egg-info folder, and then it fails while populating it (race-condition-ish unravelled by the change).

abravalheri · 2024-10-15T12:10:17Z

Hi @henryiii, in pypa/setuptools#4680 I am trying to change the code path so that the conflict between importlib.metadata and importlib_metadata is never reached. Does that solve the problems you initially reported?

You probably can test the PR specific version of setuptools by installing https:/abravalheri/setuptools/archive/refs/heads/issue-pyproject-hooks-206-take2.zip

henryiii mentioned this issue Oct 6, 2024

tests: avoid pyproject-hooks 1.2 on tests for now pypa/build#823

Closed

pradyunsg mentioned this issue Oct 6, 2024

Improve interop with importlib.metadata: variation without importlib_metadata #199

Merged

abravalheri mentioned this issue Oct 7, 2024

Avoid adding an empty/incomplete .egg-info to sys.path while traversing importlib.metadata.entry_points pypa/setuptools#4670

Closed

2 tasks

notatallshaw mentioned this issue Oct 13, 2024

Vendoring bumps for 24.3 pypa/pip#13006

Open

henryiii mentioned this issue Oct 15, 2024

Revert "tests: fix under pyproject-hooks 1.2" pypa/build#829

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Version 1.2 breaks pypa/build on Python 3.8 and 3.9 #206

Version 1.2 breaks pypa/build on Python 3.8 and 3.9 #206

henryiii commented Oct 6, 2024 •

edited

Loading

layday commented Oct 6, 2024

pradyunsg commented Oct 6, 2024

pradyunsg commented Oct 6, 2024

pfmoore commented Oct 6, 2024

takluyver commented Oct 6, 2024

layday commented Oct 6, 2024 via email

henryiii commented Oct 6, 2024 •

edited

Loading

nanonyme commented Oct 6, 2024 •

edited

Loading

henryiii commented Oct 6, 2024 •

edited

Loading

jaraco commented Oct 6, 2024

jaraco commented Oct 6, 2024

abravalheri commented Oct 6, 2024 •

edited

Loading

abravalheri commented Oct 7, 2024 •

edited

Loading

abravalheri commented Oct 15, 2024 •

edited

Loading

Version 1.2 breaks pypa/build on Python 3.8 and 3.9 #206

Version 1.2 breaks pypa/build on Python 3.8 and 3.9 #206

Comments

henryiii commented Oct 6, 2024 • edited Loading

layday commented Oct 6, 2024

pradyunsg commented Oct 6, 2024

pradyunsg commented Oct 6, 2024

pfmoore commented Oct 6, 2024

takluyver commented Oct 6, 2024

layday commented Oct 6, 2024 via email

henryiii commented Oct 6, 2024 • edited Loading

nanonyme commented Oct 6, 2024 • edited Loading

henryiii commented Oct 6, 2024 • edited Loading

jaraco commented Oct 6, 2024

jaraco commented Oct 6, 2024

abravalheri commented Oct 6, 2024 • edited Loading

abravalheri commented Oct 7, 2024 • edited Loading

abravalheri commented Oct 15, 2024 • edited Loading

henryiii commented Oct 6, 2024 •

edited

Loading

henryiii commented Oct 6, 2024 •

edited

Loading

nanonyme commented Oct 6, 2024 •

edited

Loading

henryiii commented Oct 6, 2024 •

edited

Loading

abravalheri commented Oct 6, 2024 •

edited

Loading

abravalheri commented Oct 7, 2024 •

edited

Loading

abravalheri commented Oct 15, 2024 •

edited

Loading