Recursive extra dependencies don't work for a private package

[zbynekwinkler]

Installing private package (=package not published on pypi) is possible with pip using pip install -e .[dev]. It respects extra dependencies and the extra dependencies can even reference themselves (see the example bellow). However compiling such a project to get requirements.txt fails.

Environment Versions

1. OS Type: ubuntu 18.04
2. Python version: $ python -V: Python 3.8.0
3. pip version: $ pip --version: pip 22.2.2
4. pip-tools version: $ pip-compile --version: version 6.8.0

Steps to replicate

Minimal pyproject.toml:

[project]
name = "privateproject"
version = "0.0.1"
dependencies = [
    "tomli",
]

[project.optional-dependencies]
test = [
    "pep517",
]
dev = [
    "privateproject[test]",
]

1. save minimal pyproject.toml
2. run python -m piptools compile --extra dev pyproject.toml

Expected result

The requirements.txt is generated containing tomli and pep517.

Actual result

Could not find a version that matches privateproject[test] (from privateproject (pyproject.toml))
No versions found
Was https://pypi.org/simple reachable?

[zbynekwinkler]

It seems pip-tools compile does not source the dependencies from the current pyproject.toml being processed but from the last published package version on pypi - if so, that could be a problem also for public packages.

[hynek]

Please add the [build-system] part that you used too.

[zbynekwinkler]

The above pyproject.toml file was used to get the error message (=no build-system set). In my project I use the following:

[build-system]
requires = ["setuptools"]
build-backend = "setuptools.build_meta"

[hynek]

Wait, is your project an actual Python package? Like do you install it with pip install -e .?

Because 100% of my work projects don't exist on PyPI, but they must be proper packages if you want to use packaging-based tools.

[zbynekwinkler]

Wait, is your project an actual Python package? Like do you install it with pip install -e .?

Yes, I do install it and it works. That is what I meant by:

Installing private package (=package not published on pypi) is possible with pip using pip install -e .[dev].

Since pip does install my package I figured pip-tools could be able compile the dependencies. Then I just trimmed the pyproject.toml to cut the parts not related with the issue while still getting the same error message while compiling. The result might no longer be installable by pip, I haven't checked that.

[atugushev]

Self-dependencies are not supported by pip-tools for now. Fun fact: the feature was unintentionally added in pip==21.2. I've also tried this with pip-compile --resolver backtracking, but no luck:

pip._internal.exceptions.DistributionNotFound: No matching distribution found for privateproject[test]

[hynek]

I think it would be a really nice feature to add for packages (ie.not apps) that want to pin their dependencies for CI stability but don't want to use poetry.

Are there technical hurdles or is it a matter of someone finding time to do it?

[atugushev]

Are there technical hurdles or is it a matter of someone finding time to do it?

Yeah, the latter one.

[q0w]

In pip it works because pip can detect that '.[dev]' contains a path, so it makes InstallRequirement with Link, and then the backtracking resolver makes from this InstallRequirement a LinkCandidate....
So you can do something like:

            package_name = metadata.get_all('Name')[0]
            comes_from = f"{metadata.get_all('Name')[0]} ({src_file})"
            def _f(req):
                if package_name in req:
                    req = req.replace(package_name, ".")
                return install_req_from_line(req, comes_from)
            constraints.extend(
                [
                    _f(req)
                    for req in metadata.get_all("Requires-Dist") or []
                ]
            )

and it will produce this output

#
# This file is autogenerated by pip-compile with Python 3.10
# by the following command:
#
#    pip-compile --extra=dev --resolver=backtracking pyproject.toml
#
pep517==0.13.0
    # via privateproject
privateproject @ file:///tmp/t
    # via privateproject (pyproject.toml)
tomli==2.0.1
    # via
    #   pep517
    #   privateproject
    #   privateproject (pyproject.toml)

@atugushev what do you think?

[atugushev]

@q0w looks good to me.

[ak-gupta]

+1 on this issue! I want to switch from setup.cfg to pyproject.toml and this feature is the only blocker I have at the moment

[atugushev]

@hynek @ak-gupta released as part of pip-tools v6.13.0!

[hynek]

Awesome thank you!

[ak-gupta]

Thank you!