Unable to use Hatch `Context Formatting` feature to format paths to local dependencies #826

dustyhorizon · 2024-03-03T17:19:55Z

Steps to Reproduce

Configure the pyproject.toml with the hatch build system i.e.

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[tool.hatch.metadata]
allow-direct-references = true

Add a local reference to another project using Hatch's Context Formatting feature i.e.

[project]
...
dependencies = ["somepkg @ file:///{root:parent}/otherpkg"]

Run rye sync

Expected Result

rye sync should complete successfully without errors and with the respective lock files created.

Actual Result

When working with a local dependency "util @ file:///{root:parent}/util"

Without uv:

❯ rye sync
Reusing already existing virtualenv
Generating production lockfile: /path/to/my/project/python/lib/repoutil/requirements.lock
Traceback (most recent call last):
  File "/home/user/.rye/pip-tools/[email protected]/bin/pip-compile", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File "/home/user/.rye/pip-tools/[email protected]/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.rye/pip-tools/[email protected]/lib/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/home/user/.rye/pip-tools/[email protected]/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.rye/pip-tools/[email protected]/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.rye/pip-tools/[email protected]/lib/python3.11/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.rye/pip-tools/[email protected]/lib/python3.11/site-packages/piptools/scripts/compile.py", line 659, in cli
    results = resolver.resolve(max_rounds=max_rounds)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.rye/pip-tools/[email protected]/lib/python3.11/site-packages/piptools/resolver.py", line 604, in resolve
    is_resolved = self._do_resolve(
                  ^^^^^^^^^^^^^^^^^
  File "/home/user/.rye/pip-tools/[email protected]/lib/python3.11/site-packages/piptools/resolver.py", line 636, in _do_resolve
    resolver.resolve(
  File "/home/user/.rye/pip-tools/[email protected]/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 76, in resolve
    collected = self.factory.collect_root_requirements(root_reqs)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.rye/pip-tools/[email protected]/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 534, in collect_root_requirements
    reqs = list(
           ^^^^^
  File "/home/user/.rye/pip-tools/[email protected]/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 490, in _make_requirements_from_install_req
    cand = self._make_base_candidate_from_link(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.rye/pip-tools/[email protected]/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 228, in _make_base_candidate_from_link
    self._link_candidate_cache[link] = LinkCandidate(
                                       ^^^^^^^^^^^^^^
  File "/home/user/.rye/pip-tools/[email protected]/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 293, in __init__
    super().__init__(
  File "/home/user/.rye/pip-tools/[email protected]/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 156, in __init__
    self.dist = self._prepare()
                ^^^^^^^^^^^^^^^
  File "/home/user/.rye/pip-tools/[email protected]/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 225, in _prepare
    dist = self._prepare_distribution()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.rye/pip-tools/[email protected]/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 304, in _prepare_distribution
    return preparer.prepare_linked_requirement(self._ireq, parallel_builds=True)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.rye/pip-tools/[email protected]/lib/python3.11/site-packages/pip/_internal/operations/prepare.py", line 525, in prepare_linked_requirement
    return self._prepare_linked_requirement(req, parallel_builds)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.rye/pip-tools/[email protected]/lib/python3.11/site-packages/pip/_internal/operations/prepare.py", line 596, in _prepare_linked_requirement
    local_file = unpack_url(
                 ^^^^^^^^^^^
  File "/home/user/.rye/pip-tools/[email protected]/lib/python3.11/site-packages/pip/_internal/operations/prepare.py", line 178, in unpack_url
    unpack_file(file.path, location, file.content_type)
  File "/home/user/.rye/pip-tools/[email protected]/lib/python3.11/site-packages/pip/_internal/utils/unpacking.py", line 243, in unpack_file
    or tarfile.is_tarfile(filename)
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.rye/py/[email protected]/install/lib/python3.11/tarfile.py", line 2793, in is_tarfile
    t = open(name)
        ^^^^^^^^^^
  File "/home/user/.rye/py/[email protected]/install/lib/python3.11/tarfile.py", line 1805, in open
    return func(name, "r", fileobj, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.rye/py/[email protected]/install/lib/python3.11/tarfile.py", line 1871, in gzopen
    fileobj = GzipFile(name, mode + "b", compresslevel, fileobj)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.rye/py/[email protected]/install/lib/python3.11/gzip.py", line 174, in __init__
    fileobj = self.myfileobj = builtins.open(filename, mode or 'rb')
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/{root:parent}/util'
error: could not write production lockfile for workspace

With uv:

❯ rye sync
Reusing already existing virtualenv
Generating production lockfile: /path/to/my/project/python/lib/repoutil/requirements.lock
error: Requirements contain conflicting URLs for package `util`:
- file:///{root:parent}/util
- file:////path/to/my/project/python/lib/util
error: could not write production lockfile for workspace

Caused by:
    failed to generate lockfile

Do note that file:///{root:parent}/util, when resolved by Hatch, resolves to file:////path/to/my/project/python/lib/util. Seems like Rye is checking the URLs before being resolved properly by the build system?

Version Info

❯ rye --version
rye 0.27.0
commit: 0.27.0 (1980-01-01)
platform: linux (x86_64)
self-python: [email protected]
symlink support: true
uv enabled: true

Stacktrace

No response

The text was updated successfully, but these errors were encountered:

dustyhorizon · 2024-03-03T17:37:38Z

Possibly related to #800

mitsuhiko · 2024-03-03T18:18:29Z

That is unrelated. The issue here is from my perspective that hatch is doing something that the standard does not really suggest should work. How should a tool like rye understand which proprietary extensions are used in that dependency array?

dustyhorizon · 2024-03-04T02:02:51Z

The odd thing is that this works if the package using this local dependency and exact syntax works when being a member of the rye workspace when specified at the root of the repository i.e. /path/to/my/project/pyproject.toml

Instead of "rooting" this at the root of the repository, I am trying to run rye sync from within the package itself just to reduce the scope. I am working with a monorepo and am trying to see if I can do away with constantly changing the rye workspace members.

Outputs to follow later today if I were to do away with this syntax and use ${PROJECT_ROOT} instead.

dustyhorizon · 2024-03-04T13:44:32Z

Oddly enough, the file:///{root:parent}/util syntax works if there is a pyproject.toml in the parent directory functioning as the workspace root. The above errors appears when the workspace root is referring to a package in a sibling folder i.e. members = ["../sibling"]

Given the folders a/package_a and a/package_b, if a/package_a/pyproject.toml specifies a/package_b as a workspace member i.e. members = ["../package_b"] then the abovementioned errors will appear.

Given the same set of folders, if a/pyproject.toml specifies both a/package_a and a/package_b as workspace members then rye lock runs without errors.

mitsuhiko · 2024-03-06T20:32:38Z

That's most likely because in that case hatch builds a wheel itself and the parent package just observes the finalized wheel and it's metadata, never reading the pyproject.toml. Unfortunately I don't think there is much we can do. From where I stand, hatch encourages something that is not standardized.

ofek · 2024-03-16T14:19:47Z

I wouldn't say encourages but rather that syntax allows for certain use cases that are otherwise difficult or impossible to represent in a correct and cross-platform fashion.

pvardanis · 2024-05-08T13:35:35Z

if rye resolves to the absolute local path, how's it possible then to add local dependencies that can be reproducable in other machines as well? For example in a docker image, passing the local dependency as a build context is straightforward however the path won't be the same

bluss · 2024-05-08T15:40:50Z

@pvardanis

Rye's first response to this is workspaces. That's how rye solves "local dependencies" without having to specify any paths, then all the packages have to share a workspace and no absolute paths end up in the lockfile.

Then there is incomplete support for relative paths dependencies in some ways for the pdm build backend (rye init option), where no absolute paths end up in the lockfile. The more general case for how path deps should work is pending work and design I think.

pvardanis · 2024-05-08T15:49:52Z

@bluss I understand, workspaces don't solve though the issue with workspace packages running in separate envs (whenever there's a specific reason to do so) to avoid dependency conflicts. See this comment: #1023 (comment)

ofek · 2024-05-08T15:55:03Z

Workspaces will be coming to Hatch next month.

mitsuhiko closed this as completed Mar 6, 2024

charliermarsh mentioned this issue Mar 16, 2024

uv fails to parse pyproject.toml when using Hatchling's {root:uri} for dependencies astral-sh/uv#2475

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unable to use Hatch `Context Formatting` feature to format paths to local dependencies #826

Unable to use Hatch `Context Formatting` feature to format paths to local dependencies #826

dustyhorizon commented Mar 3, 2024 •

edited

Loading

dustyhorizon commented Mar 3, 2024

mitsuhiko commented Mar 3, 2024

dustyhorizon commented Mar 4, 2024 •

edited

Loading

dustyhorizon commented Mar 4, 2024 •

edited

Loading

mitsuhiko commented Mar 6, 2024

ofek commented Mar 16, 2024

pvardanis commented May 8, 2024 •

edited

Loading

bluss commented May 8, 2024 •

edited

Loading

pvardanis commented May 8, 2024

ofek commented May 8, 2024

Unable to use Hatch Context Formatting feature to format paths to local dependencies #826

Unable to use Hatch Context Formatting feature to format paths to local dependencies #826

Comments

dustyhorizon commented Mar 3, 2024 • edited Loading

Steps to Reproduce

Expected Result

Actual Result

Version Info

Stacktrace

dustyhorizon commented Mar 3, 2024

mitsuhiko commented Mar 3, 2024

dustyhorizon commented Mar 4, 2024 • edited Loading

dustyhorizon commented Mar 4, 2024 • edited Loading

mitsuhiko commented Mar 6, 2024

ofek commented Mar 16, 2024

pvardanis commented May 8, 2024 • edited Loading

bluss commented May 8, 2024 • edited Loading

pvardanis commented May 8, 2024

ofek commented May 8, 2024

Unable to use Hatch `Context Formatting` feature to format paths to local dependencies #826

Unable to use Hatch `Context Formatting` feature to format paths to local dependencies #826

dustyhorizon commented Mar 3, 2024 •

edited

Loading

dustyhorizon commented Mar 4, 2024 •

edited

Loading

dustyhorizon commented Mar 4, 2024 •

edited

Loading

pvardanis commented May 8, 2024 •

edited

Loading

bluss commented May 8, 2024 •

edited

Loading