Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ignore hashes for version control repositories #994

Open
a666 opened this issue Nov 20, 2019 · 13 comments
Open

Ignore hashes for version control repositories #994

a666 opened this issue Nov 20, 2019 · 13 comments
Labels
pip Related to pip vcs Related to VCS requirements

Comments

@a666
Copy link

a666 commented Nov 20, 2019

What's the problem this feature will solve?

Right now we are having the following situation.

Having a requirements.in like:

# requirements.in
django
git+ssh://[email protected]/[email protected]#egg=private_app

and doing:

$ pip-compile --generate-hashes -o requirements.txt requirements.in && pip-sync requirements.txt

gives us the error:

ERROR: Can't verify hashes for these requirements because we don't have a way to hash version control repositories:
    private_app from git+ssh://****@private.server/[email protected]#egg=private_app 
    (from -r /tmp/tmpii51x619 (line 217))

Replacing our repo with -e git+ssh://[email protected]/[email protected]#egg=private_app changes the error to

ERROR: The editable requirement posgrado_catalogs from 
git+ssh://****@private.server/[email protected]#egg=private_app
(from -r /tmp/tmp317kk_qv (line 172)) cannot be installed when requiring hashes, 
because there is no single file to hash.

Describe the solution you'd like

It would be ideal for version control repositories to skip hashing (since access and integrity is handled differently) while keeping it for every other package.

Alternative Solutions

pipenv (ugh) seems to do it.

@atugushev
Copy link
Member

atugushev commented Nov 20, 2019

Hello @a666,

Thanks for the issue! I believe this should be implemented on pip side. See related issues:

However, there are possible solutions:

@atugushev atugushev added the pip Related to pip label Nov 20, 2019
@a666
Copy link
Author

a666 commented Nov 20, 2019

Thank you @atugushev for the quick answer.

I forgot to precise that we are using your first solution for the time being.

@atugushev atugushev added the vcs Related to VCS requirements label Jun 19, 2020
@cjerdonek
Copy link

cjerdonek commented Feb 24, 2021

Hi @atugushev, what did you mean by this?

  • split requirements files for hashable and non-hashable packages.

Did you mean split the requirements.in file? Or did you mean split the requirements.txt file output by pip-compile (e.g. into hashable and non-hashable requirements)? (The latter could presumably be done by parsing the output, though it's not so elegant.)

By the way, it seems like if #333 were implemented, that would be another possible solution. If one knows certain VCS requirements are included in a requirements.in file, then the names of those packages could also be passed in via #333's --exclude option to exclude those lines from the output.

@atugushev
Copy link
Member

Did you mean split the requirements.in file?

Hello @cjerdonek! That's exactly what I mean. Yeah, --exclude seems like a good solution.

@cjerdonek
Copy link

Thanks, @atugushev! However, won't splitting the requirements.in file into VCS and non-VCS requirements mean that the dependencies of the VCS requirements (which can be hashable) won't get the benefit of the hashes, because those dependencies would be part of the requirements.txt file corresponding to the VCS requirements.in? Or am I missing part of your suggestion?

@atugushev
Copy link
Member

@cjerdonek, yes, you are right.

@cjerdonek
Copy link

Okay, thanks a lot for confirming.

@sergeyklay
Copy link
Member

sergeyklay commented Jun 5, 2021

Hello,

Not sure if this is the right place, but I'd like to show you my case so that you have more information about the issues that arise. Most of the code has been omitted to show only the gist:

tox.ini

[tox]
minversion = 3.22
envlist = py{37,38,39,310}

[testenv]
extras = testing
deps =
    -rrequirements.txt
commands =
    coverage erase
    coverage run -m pytest {posargs}

requirements.in

django

# ...

Command to compile requirements.txt file

pip-compile --generate-hashes --output-file=requirements.txt requirements.in

requirements.txt

# line line 11
asgiref==3.3.4 \
    --hash=sha256:92906c611ce6c967347bbfea733f13d6313901d54dcca88195eaeb52b2a8e8ee \
    --hash=sha256:d1216dfbdfb63826470995d31caed36225dcaf34f182e0fa257a4dd9e86f1b78
    # via django

# ...

Relevant part of setup.py

EXTRAS_REQUIRE = {
    'testing': [
        'pytest>=6.2.0',
        'pytest-cov>=2.11.1',
        'pytest-django>=4.2.0', 
        'factory-boy>=3.2.0',
        'faker>=8.1.0',
    ],
}

# ...

if __name__ == '__main__':
    setup(
        # ...

        extras_require=EXTRAS_REQUIRE,
    )

Command to install deps and run tests

tox

Output from GitHub Actions

Collecting typing-extensions
ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
typing-extensions from https://files.pythonhosted.org/packages/2e/35/6c4fff5ab443b57116cb1aad46421fb719bed2825664e8fe77d66d99bcbc/typing_extensions-3.10.0.0-py3-none-any.whl#sha256=779383f6086d90c99ae41cf0ff39aac8a7937a9283ce0a414e5dd782f4c94a84
(from asgiref==3.3.4->-r requirements.txt (line 11))

=================================== log end ====================================

ERROR: could not install deps [-rrequirements.txt]; v = InvocationError('/home/runner/work/branch/branch/.tox/py37/bin/python -m pip install -rrequirements.txt', 1)
___________________________________ summary ____________________________________

ERROR: py37: could not install deps [-rrequirements.txt]; v = InvocationError('/home/runner/work/branch/branch/.tox/py37/bin/python -m pip install -rrequirements.txt', 1)


This issue occurs only for Python 3.7. Possible due this part of setup.cfg from asgiref repo:

[options]
python_requires = >=3.6
packages = find:
include_package_data = true
install_requires =
    typing_extensions; python_version < "3.8"
zip_safe = false

Possible related issues:


Let me know if I can provide more information.

sergeyklay added a commit to sergeyklay/branch that referenced this issue Jun 5, 2021
sergeyklay added a commit to sergeyklay/gstore that referenced this issue Jun 6, 2021
sergeyklay added a commit to sergeyklay/gstore that referenced this issue Jun 6, 2021
@nstylo
Copy link

nstylo commented Dec 17, 2021

I'd also like to see this feature. Currently we have some local dependencies checked into VCS (with git lfs) and I'd like to ignore those dependencies when running pip-compile --generate-hashes

@GabrielDumbrava
Copy link

Thank you @atugushev for the quick answer.

I forgot to precise that we are using your first solution for the time being.

Hi!
Can you please give me an example of your requirements.in file that works for you? I can't make it work. Here's mine:

requirements.in

pillow
-e git+https:/hvlads/django-ckeditor-5/archive/073b40bbf506d084f9b20788e6a17187001b65ae.zip#egg=django-ckeditor-5

I get an error when pip-compile is trying to clone:

$pip-compile --generate-hashes requirements.in

ERROR:pip.subprocessor:[present-diagnostic] git clone --filter=blob:none --quiet https:/hvlads/django-ckeditor-5/archive/073b40bbf506d084f9b20788e6a17187001b65ae.zip /[SOME_PATH]/src/django-ckeditor-5 exited with 128
Traceback (most recent call last):

If I ran the clone myself:

$git clone --filter=blob:none --quiet https:/hvlads/django-ckeditor-5/archive/073b40bbf506d084f9b20788e6a17187001b65ae.zip

remote: Not Found
fatal: repository 'https:/hvlads/django-ckeditor-5/archive/073b40bbf506d084f9b20788e6a17187001b65ae.zip/' not found

Getting the archive with wget works:

$ wget https:/hvlads/django-ckeditor-5/archive/073b40bbf506d084f9b20788e6a17187001b65ae.zip
--2022-06-06 10:55:09--  https:/hvlads/django-ckeditor-5/archive/073b40bbf506d084f9b20788e6a17187001b65ae.zip
Resolving github.com (github.com)... 140.82.121.4
Connecting to github.com (github.com)|140.82.121.4|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://codeload.github.com/hvlads/django-ckeditor-5/zip/073b40bbf506d084f9b20788e6a17187001b65ae [following]
--2022-06-06 10:55:10--  https://codeload.github.com/hvlads/django-ckeditor-5/zip/073b40bbf506d084f9b20788e6a17187001b65ae
Resolving codeload.github.com (codeload.github.com)... 140.82.121.9
Connecting to codeload.github.com (codeload.github.com)|140.82.121.9|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [application/zip]
Saving to: ‘073b40bbf506d084f9b20788e6a17187001b65ae.zip’

073b40bbf506d084f9b20788e6a17187001b65ae.zip                                      [  <=>                                                                                                                                                                                             ]   1,63M  2,46MB/s    in 0,7s    

2022-06-06 10:55:11 (2,46 MB/s) - ‘073b40bbf506d084f9b20788e6a17187001b65ae.zip’ saved [1708665]

If I replace the url with the redirected one https://codeload.github.com/hvlads/django-ckeditor-5/zip/073b40bbf506d084f9b20788e6a17187001b65ae I get the same result.

@AndydeCleyre
Copy link
Contributor

@GabrielDumbrava

This link isn't actually a git protocol thing, and not really editable, so try this change:

- -e git+https:/hvlads/djan...
+ https:/hvlads/djan...

@jedie
Copy link

jedie commented Nov 20, 2022

I have a similar error like above #994 (comment)

ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
​ typing-extensions from https://files.pythonhosted.org/packages/2e/35/6c4fff5ab443b57116cb1aad46421fb719bed2825664e8fe77d66d99bcbc/typing_extensions-3.10.0.0-py3-none-any.whl#sha256=779383f6086d90c99ae41cf0ff39aac8a7937a9283ce0a414e5dd782f4c94a84
(from asgiref==3.3.4->-r requirements.txt (line 11))

In my case, it's also GitHub Actions and happens only with Python 3.9 (with 3.10 and 3.11 everything is fine). Looks like:

Collecting typing-extensions>=3.10.0.0
ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
    typing-extensions>=3.10.0.0 from https://files.pythonhosted.org/packages/0b/8e/f1a0a5a76cfef77e1eb6004cb49e5f8d72634da638420b9ea492ce8305e8/typing_extensions-4.4.0-py3-none-any.whl (from black==22.10.0->-r /tmp/tmphawi3o6_ (line 7))

Complete CI run: https:/jedie/cookiecutter_templates/actions/runs/3509450818/jobs/5878554915

black defines typing_extensions in pyproject.toml / dependencies with:

"typing_extensions>=3.10.0.0; python_version < '3.10'",

See: https:/psf/black/blob/27932494bcefac03497dd92dcf0c59a04c10d757/pyproject.toml#L65-L73

@atugushev
Copy link
Member

atugushev commented Jul 9, 2023

It seems like hash-checking mode for VCS URLs would be supported soon:

After the PR is merged, some work would be required on the pip-tools side to hide the warning.

jedie pushed a commit to jedie/cookiecutter_templates that referenced this issue Aug 25, 2024
jedie added a commit to jedie/manageprojects that referenced this issue Aug 29, 2024
jedie added a commit to jedie/manageprojects that referenced this issue Aug 29, 2024
Actionb added a commit to Actionb/MIZDB that referenced this issue Sep 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
pip Related to pip vcs Related to VCS requirements
Projects
None yet
Development

No branches or pull requests

8 participants