Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add GraalPy #1520

Merged
merged 1 commit into from
Jun 22, 2024
Merged

Add GraalPy #1520

merged 1 commit into from
Jun 22, 2024

Conversation

mayeut
Copy link
Member

@mayeut mayeut commented Aug 19, 2023

In order to install GraalPy in the image, one must call manylinux-interpreters ensure graalpy310-graalpy230_310_native.

closes #1509

tests/run_tests.sh Outdated Show resolved Hide resolved
@mayeut
Copy link
Member Author

mayeut commented Aug 19, 2023

@timfel,
I tried to backport most of the comments from #1518 in this PR.
We'll able to move forward once the "skip the simple project build on graalpy" is dropped.

@mayeut mayeut mentioned this pull request Aug 19, 2023
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

timfel wrote:

FYI, I added something to get the versions of graalpy to my earlier cibuildwheel PR: timfel/cibuildwheel@ace30fb#diff-2fc4993e0fc3281ad7dfa0e696c6791c9339e16180d62b6fdd05df72a217ced5R112. Not sure if we want to do something like this here to more easily update graalpy, too. I can also do that in a separate PR if/when this one goes in.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That would be a good thing to add overall but this can until this one's merged.

@timfel
Copy link
Contributor

timfel commented Oct 6, 2023

@mayeut we have released graalpy 23.1, this should now produce the correct wheel tags and pass the tests. Can we update this PR?

docker/build_scripts/python_versions.json Outdated Show resolved Hide resolved
docker/build_scripts/python_versions.json Outdated Show resolved Hide resolved
docker/build_scripts/python_versions.json Outdated Show resolved Hide resolved
docker/build_scripts/python_versions.json Outdated Show resolved Hide resolved
docker/build_scripts/python_versions.json Outdated Show resolved Hide resolved
docker/build_scripts/python_versions.json Outdated Show resolved Hide resolved
docker/build_scripts/python_versions.json Outdated Show resolved Hide resolved
@mayeut
Copy link
Member Author

mayeut commented Oct 7, 2023

@timfel, I updated the PR but the tests are not passing. pip 23.3 should fix the issue.

@timfel
Copy link
Contributor

timfel commented Oct 8, 2023

@timfel, I updated the PR but the tests are not passing. pip 23.3 should fix the issue.

Ah yes :( We backported the fix into pip 22.2 whl that we ship, but of course on manylinux we use the latest pip

@timfel
Copy link
Contributor

timfel commented Oct 16, 2023

@timfel, I updated the PR but the tests are not passing. pip 23.3 should fix the issue.

23.3 is out, should we rerun the ci?

@mayeut
Copy link
Member Author

mayeut commented Oct 21, 2023

The update of packaging has been moved to 24.1 by the pip team.

@timfel
Copy link
Contributor

timfel commented May 9, 2024

Thank you for the update Matthieu 🙏 Let me know if there's anything I need to help with from the GraalPy side 🙂

@mayeut
Copy link
Member Author

mayeut commented May 10, 2024

@timfel,

It seems the latest beta of pip now works properly with GraalPy so this will most likely be ready in a couple weeks.

Let me know if there's anything I need to help with from the GraalPy side

If you can have a look at how to get automated updates in tools/update_interpreters_download.py, it would be nice to have it. Thanks.

@henryiii
Copy link
Contributor

pip 24.1 is out!

@msimacek
Copy link

Hi @mayeut,
Here's the changes for tools/update_interpreters_download.py:

index 094653e..ce28e25 100644
--- a/tools/update_interpreters_download.py
+++ b/tools/update_interpreters_download.py
@@ -2,6 +2,7 @@ from __future__ import annotations
 
 import argparse
 import json
+import re
 import subprocess
 from hashlib import sha256
 from pathlib import Path
@@ -82,6 +83,69 @@ def update_pypy_versions(versions, updates):
             )
 
 
+def update_graalpy_version(releases, graalpy_spec, tag, arch, version_dict, updates):
+    graalpy_arch = {"x86_64": "amd64"}.get(arch, arch)
+    current_version = None
+    if "version" in version_dict:
+        current_version = Version(version_dict["version"])
+    for r in releases:
+        version = Version(r['tag_name'].split('-')[1])
+        if current_version is not None and current_version >= version:
+            continue
+        if not graalpy_spec.contains(version):
+            continue
+        message = f"updating {tag} {arch} to {version}"
+        print(message)
+        for asset in r['assets']:
+            if asset['name'] == f'graalpy-{version}-linux-{graalpy_arch}.tar.gz':
+                break
+        response = requests.get(asset["browser_download_url"], stream=True)
+        response.raise_for_status()
+        sha256sum = sha256()
+        for chunk in response.iter_content(chunk_size=1024 * 4):
+            sha256sum.update(chunk)
+        version_dict["version"] = str(version)
+        version_dict["download_url"] = asset["browser_download_url"]
+        version_dict["sha256"] = sha256sum.hexdigest()
+        updates.append(message)
+        break
+
+
+def get_next_page_link(response):
+    link = response.headers.get('link')
+    if link:
+        for part in re.split(r'\s*,\s*', link):
+            split = re.split(r'\s*;\s*', part)
+            url = split[0][1:-1]
+            for param in split[1:]:
+                if re.match(r'rel="?next"?', param):
+                    return url
+
+
+def update_graalpy_versions(versions, updates):
+    releases = []
+    url = "https://api.github.com/repos/oracle/graalpython/releases"
+    while url:
+        response = requests.get(url)
+        response.raise_for_status()
+        releases += response.json()
+        url = get_next_page_link(response)
+    for tag in versions:
+        if not tag.startswith("graalpy"):
+            continue
+        _, abi_tag = tag.split("-")
+        graalpy_ver, _, _ = abi_tag.split("_")
+        assert graalpy_ver.startswith("graalpy")
+        graalpy_ver = graalpy_ver[len("graalpy"):]
+        graalpy_major = int(graalpy_ver[:2])
+        graalpy_minor = int(graalpy_ver[2:])
+        graalpy_spec = Specifier(f"=={graalpy_major}.{graalpy_minor}.*")
+        for arch in versions[tag]:
+            update_graalpy_version(
+                releases, graalpy_spec, tag, arch, versions[tag][arch], updates
+            )
+
+
 def main():
     parser = argparse.ArgumentParser()
     parser.add_argument("--dry-run", dest="dry_run", action="store_true", help="dry run")
@@ -89,6 +153,7 @@ def main():
     versions = json.loads(PYTHON_VERSIONS.read_text())
     updates = []
     update_pypy_versions(versions, updates)
+    update_graalpy_versions(versions, updates)
     if not args.dry_run:
         PYTHON_VERSIONS.write_text(json.dumps(versions, indent=2))
         if updates:

@mayeut mayeut marked this pull request as ready for review June 22, 2024 08:43
@mayeut
Copy link
Member Author

mayeut commented Jun 22, 2024

This is now good to merge !
Thanks everyone.

@mayeut mayeut merged commit 8811efb into pypa:main Jun 22, 2024
9 checks passed
@mayeut mayeut deleted the add-graalpy branch June 22, 2024 12:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants