Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WORKAROUND] Unable to build PyYAML < 6.0.1 from source or sdist #736

Open
nitzmahone opened this issue Jul 26, 2023 · 11 comments
Open

[WORKAROUND] Unable to build PyYAML < 6.0.1 from source or sdist #736

nitzmahone opened this issue Jul 26, 2023 · 11 comments

Comments

@nitzmahone
Copy link
Member

In the instructions below, replace pip as needed with a pip invocation matching your target Python environment (eg, pip3, /usr/local/bin/python3.11 -m pip).

Pre-seed wheel cache (recommended)

This solution pre-seeds pip's wheel cache with a locally-built PyYAML wheel, accessible to any subsequent installation using the same pip cache.

# create a constraint file that limits the Cython version to one that should work
echo 'Cython < 3.0' > /tmp/constraint.txt

# seed pip's local wheel cache with a PyYAML wheel
PIP_CONSTRAINT=/tmp/constraint.txt pip wheel PyYAML==5.4.1

# install PyYAML itself, or any other package(s) that ask for the PyYAML version you just built
pip install 'PyYAML==5.4.1'

Inline constraint (simpler, more likely to break)

This solution globally constrains the Cython version for all packages being installed by this pip invocation (including nested/child package installs), which could break other packages installed at the same time.

# create a constraint file that limits the Cython version to one that should work
echo 'Cython < 3.0' > /tmp/constraint.txt

# install PyYAML itself (or other packages that need it); any package requiring Cython will be constrained to `Cython < 3.0`
PIP_CONSTRAINT=/tmp/constraint.txt pip install 'PyYAML==5.4.1'

Background

With the release of Cython 3, all older versions of PyYAML can no longer be installed from unmodified source or sdist (ie, where a wheel is unavailable for the platform and/or Python), since the Cython version was not capped to a working version for all older PyYAML releases. For various reasons, it is untenable to release new sdists/wheels for these old PyYAML versions with the new required Cython<3.0 build dependency constraint.

pip has mostly-undocumented support for "inherited" constraints at install-time by setting the PIP_CONSTRAINT environment variable (which is inherited by child build processes, unlike the CLI --constraint arg).

Sample error output from pip install

If you're seeing an error similar to this when installing a version of PyYAML older than 6.0.1, the preceding solutions may help.

...
  error: subprocess-exited-with-error
  
  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [48 lines of output]
      running egg_info
      writing lib/PyYAML.egg-info/PKG-INFO
      writing dependency_links to lib/PyYAML.egg-info/dependency_links.txt
      writing top-level names to lib/PyYAML.egg-info/top_level.txt
      Traceback (most recent call last):

...

          raise AttributeError(attr)
      AttributeError: cython_sources
      [end of output]
@leiflinse-trivector
Copy link

In a case where 5.4.1 arrives as a dependency from requirements.txt the workaround for us looks like this. Unfortunately when we build just the wheel with PIP_CONSTRAINT, that wheel is not used when installing from requirements.txt.

echo 'Cython < 3.0' > /tmp/constraint.txt
PIP_CONSTRAINT=/tmp/constraint.txt pip install 'PyYAML==5.4.1'

pip install -r requirements.txt

Thanks for posting the workaround. It was very helpful.

@nitzmahone
Copy link
Member Author

nitzmahone commented Jul 27, 2023

@leiflinse-trivector Interesting- we (Ansible) have successfully deployed this workaround extensively throughout our CI infra on dozens of different OSs from Python 2.7-3.12 and numerous pip versions, just as described above. One caveat I purposely omitted from the description above (since I assumed people were only building PyYAML for platforms/Pythons that didn't already have a wheel): if the index server pip consults has an "applicable" PyYAML wheel, it seems to prefer the wheel on the index server to the local one. This seems to happen more if the PyYAML requirement is unconstrained by the dependent package- the presence of any exact or upper-bound constraint on PyYAML seems to make the locally-cached wheel eligible for use (eg, if the dependent package asks for PyYAML==5.4.1 or PyYAML<6)- if the dependent package doesn't specify an upper bound and is incompatible with PyYAML 6.x, it's already arguably broken anyway, but in that case, you can also use the PIP_CONSTRAINT trick to augment the broken package/requirements install with your own constraint for eg, PyYAML<6 that should cause it to use the locally-cached wheel.

@reid-harrison
Copy link

Is there a similar workaround for pipenv? Honestly this is what I'm doing but not sure I like it...

pip3 install "cython<3.0" wheel && pip3 install --no-build-isolation "pyyaml==5.4.1" && \
pipenv sync --system 

@nitzmahone
Copy link
Member Author

nitzmahone commented Jul 31, 2023

@reid-harrison No idea here- not a pipenv user. That said, all you really need is a locally-built 5.4.1 wheel that's been constrained as necessary, so why not just do the pip wheel as describe above to create the wheel, then just install the wheel that pip created in CWD with pipenv as normal?

saklar13 added a commit to QualiSystems/cloudshell-shell-automation-tests that referenced this issue Aug 2, 2023
said-moj added a commit to ministryofjustice/cla_backend that referenced this issue Aug 7, 2023
said-moj added a commit to ministryofjustice/cla_backend that referenced this issue Aug 9, 2023
said-moj added a commit to ministryofjustice/cla_backend that referenced this issue Aug 10, 2023
said-moj added a commit to ministryofjustice/cla_backend that referenced this issue Aug 16, 2023
softwarefactory-project-zuul bot pushed a commit to ansible-collections/community.aws that referenced this issue Aug 24, 2023
CI: Force PyYAML version that can be installed

SUMMARY
Currently integration tests with ansible/ansible-zuul-jobs#1816 still fail when installing some requirements:
2023-08-22 05:53:46.350742 | TASK [ansible-test : Install python requirements]
[...]

2023-08-22 05:53:53.860850 | controller | Collecting PyYAML<5.5,>=3.10 (from awscli->-r /home/zuul-worker/.ansible/collections/ansible_collections/community/aws/test-requirements.txt (line 19))
2023-08-22 05:53:53.866271 | controller |   Downloading PyYAML-5.4.1.tar.gz (175 kB)
2023-08-22 05:53:53.878160 | controller |      �������������������������������������� 175.1/175.1 kB 19.0 MB/s eta 0:00:00
2023-08-22 05:53:54.065094 | controller |   Installing build dependencies: started
2023-08-22 05:53:58.372457 | controller |   Installing build dependencies: finished with status 'done'
2023-08-22 05:53:58.373282 | controller |   Getting requirements to build wheel: started
2023-08-22 05:53:58.680407 | controller |   Getting requirements to build wheel: finished with status 'error'
2023-08-22 05:53:58.689654 | controller |   error: subprocess-exited-with-error

(https://b3e35e76f7fc3e0cf6b5-d430d94f79a1de67219002b64dcf7484.ssl.cf1.rackcdn.com/1904/63f02a9c37ef75f9dcf25159fb4033d6edf146ee/check/integration-community.aws-1/b2f8016/job-output.txt)
Tries to work around this problem by pinning the PyYAML version to 5.3.1 (see also yaml/pyyaml#736).
ISSUE TYPE

Test Pull Request

COMPONENT NAME
integration tests

Reviewed-by: Alina Buzachis
Reviewed-by: Mark Chappell
Reviewed-by: Felix Fontein <[email protected]>
Reviewed-by: Markus Bergholz <[email protected]>
@nanonyme
Copy link

nanonyme commented Apr 3, 2024

Just a warning to maintainers: PyYAML supporting Python 3.13 is supposedly going to require switching to Cython 3.

@nitzmahone
Copy link
Member Author

@nanonyme Yeah, I've been doing all my local 3.13 testing against Cython 3 with the old_build_ext stuff and it's been working fine. We'll probably end up implementing at least conditional support for that in the update that supports 3.13, since the "shred the extension build to ditch all the distutils-isms" thing is still in flux.

askmeaboutlo0m added a commit to drawpile/Drawpile that referenced this issue Apr 29, 2024
Specifically because of yaml/pyyaml#736 that causes older versions of
PyYaml to not install properly on newer versions of Python.
askmeaboutlo0m added a commit to drawpile/Drawpile that referenced this issue Apr 29, 2024
Specifically because of yaml/pyyaml#736 that causes older versions of
PyYaml to not install properly on newer versions of Python.
KeiIshidaAbel added a commit to c-3lab/ckanext-feedback that referenced this issue May 14, 2024
@Uzume
Copy link

Uzume commented May 18, 2024

@nanonyme [...] supporting Python 3.13 is supposedly going to require switching to Cython 3.

My understanding is this is due to python/cpython#104775 (CPython 3.13+) and cython/cython#5128 (Cython 0.29.36+).

@nitzmahone [...] the "shred the extension build to ditch all the distutils-isms" thing is still in flux.

Care to elucidate what is holding back the jettison of distutils APIs in #797 (but feel free to comment over there where it is probably more pertinent to the discussion)?

ravindk89 added a commit to minio/docs that referenced this issue Jun 7, 2024
yaml/pyyaml#736

Python 3.12+ is somewhat broken when trying to install fresh.

Bumping PyYaml to 6.0.1 seems to fix it. There are some warnings that we
might have to freeze at Python 3.12 until PyYaml updates to fully
support it at some indeterminate time in the future.

@djwfyi @feorlen please test w/ a clean venv (e.g. `python -m venv
testvenv` or something) and make sure that you can still `pip install -r
requirements.txt` and build w/ PyYaml 6.0.1
FlyGoat added a commit to FlyGoat/u-boot that referenced this issue Jun 29, 2024
Bump python packages to reasonable versions to fix build
error with Python 3.12.

Link: pytest-dev/pytest#11094
Link: yaml/pyyaml#736
Link: pypa/setuptools#4002
Signed-off-by: Jiaxun Yang <[email protected]>
FlyGoat added a commit to FlyGoat/u-boot that referenced this issue Jun 29, 2024
Bump python packages to reasonable versions to fix build
error with Python 3.12.

Link: pytest-dev/pytest#11094
Link: yaml/pyyaml#736
Signed-off-by: Jiaxun Yang <[email protected]>
arojasb3 pushed a commit to arojasb3/astronomer-cosmos that referenced this issue Jul 14, 2024
This PR installs apache-airflow in the hatch `pre-install-commands`
section so Airflow related dependency conflicts do not need to be
managed in the override section that has been removed.

Installing from the Airflow constraint file for versions <=2.6 fails
because PyYAML is pinned in the constraint file for those versions to
6.0.0 and [a workaround ](yaml/pyyaml#736) is
required becaused older versions of PyYAML can no longer be installed
from unmodified source or sdist with the release of Cython3.

Closes: astronomer#811
madwort added a commit to opensafely-core/job-runner that referenced this issue Sep 27, 2024
* hoping this will fix the compiling from source issue that was preventing
  @Providence-o from installing pyyaml locally on Apple Silicon
* yaml/pyyaml#736
madwort added a commit to opensafely-core/job-runner that referenced this issue Sep 30, 2024
* hoping this will fix the compiling from source issue that was preventing
  @Providence-o from installing pyyaml locally on Apple Silicon
* yaml/pyyaml#736
madwort added a commit to opensafely-core/job-runner that referenced this issue Sep 30, 2024
* hoping this will fix the compiling from source issue that was preventing
  @Providence-o from installing pyyaml locally on Apple Silicon
* yaml/pyyaml#736
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

10 participants