pipx install fails when providing constraint file with hashes

[mayeut]

Describe the bug
pipx install --pip-args=--constraint=constraints.txt auditwheel fails when constraints.txt uses hashes.
This is most likely linked to pypa/pip#9243 but, as I saw the issue with pipx, I though I'd also report this here.
The use-case for this is to have some tools installed in a reproducible manner.

How to reproduce

[root@44456984b498 ~]# cat constraints.txt 
auditwheel==4.0.0 \
    --hash=sha256:03a079fe273f42336acdb5953ff5ce7578f93ca6a832b16c835fe337a1e2bd4a \
    --hash=sha256:96927695ddf27b4edb67291e326908d64ffe272b8a42b9504f283e7ae5ebbc14
    # via -r requirements-tools.in
pyelftools==0.27 \
    --hash=sha256:5609aa6da1123fccfae2e8431a67b4146aa7fad5b3889f808df12b110f230937 \
    --hash=sha256:cde854e662774c5457d688ca41615f6594187ba7067af101232df889a6b7a66b
    # via auditwheel

[root@44456984b498 ~]# pipx install --pip-args=--constraint=constraints.txt --verbose auditwheel
pipx >(setup:717): pipx version is 0.16.3
pipx >(setup:718): Default python interpreter is '/opt/_internal/tools/bin/python'
pipx >(package_name_from_spec:323): Determined package name: auditwheel
pipx >(package_name_from_spec:324): Package name determined in 0.0s
creating virtual environment...
pipx >(run_subprocess:135): running /opt/_internal/tools/bin/python -m venv --without-pip /root/.local/pipx/venvs/auditwheel
pipx >(run_subprocess:135): running /root/.local/pipx/venvs/auditwheel/bin/python -c import sysconfig; print(sysconfig.get_path('purelib'))
pipx >(run_subprocess:135): running /root/.local/pipx/shared/bin/python -c import sysconfig; print(sysconfig.get_path('purelib'))
pipx >(run_subprocess:135): running /root/.local/pipx/venvs/auditwheel/bin/python --version
pipx >(_parsed_package_to_package_or_url:128): cleaned package spec: auditwheel
installing auditwheel...
pipx >(run_subprocess:135): running /root/.local/pipx/venvs/auditwheel/bin/python -m pip install --constraint=constraints.txt auditwheel
pipx >(subprocess_post_check_handle_pip_error:297): '/root/.local/pipx/venvs/auditwheel/bin/python -m pip install --constraint=constraints.txt auditwheel' failed
pipx >(subprocess_post_check_handle_pip_error:314): Fatal error from pip prevented installation. Full pip output in file:
    /root/.local/pipx/logs/cmd_2021-07-04_12.33.00_pip_errors.log

pipx >(analyze_pip_output:272): pip seemed to fail to build package:
    auditwheel

pipx >(rmdir:43): removing directory /root/.local/pipx/venvs/auditwheel
Error installing auditwheel.

[root@44456984b498 ~]# cat /root/.local/pipx/logs/cmd_2021-07-04_12.33.00_pip_errors.log
PIP STDOUT
----------
Collecting auditwheel

PIP STDERR
----------
ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
    auditwheel from https://files.pythonhosted.org/packages/90/e1/4af4cbdef79dde083498ae408d629ce5d16728177b970c92344f433257a0/auditwheel-4.0.0-py3-none-any.whl#sha256=96927695ddf27b4edb67291e326908d64ffe272b8a42b9504f283e7ae5ebbc14

Expected behavior
package installed properly, respecting constraints

[cs01]

Just a guess, but have you tried passing ‘—spec auditwheel==4.0.0 auditwheel’ to pipx?

I am wondering if this is the issue (it’s not pinned with ==):

pip install --constraint=constraints.txt auditwheel

[uranusjr]

Looks like pypa/pip#9020 (comment)

I think you’ll need to tell pipx to pass both pinned versions of auditwheel and pyelftools. Not sure how that can be done though. Is using -r not an option?

[mayeut]

Is using -r not an option?

the -r workaround works fine when using only pip but pipx need to know the package we want binaries for (unless there's a magic option somewhere ?) and thus it's added after the "-r"

[root@44456984b498 ~]# pipx install --pip-args=--requirement=constraints.txt --verbose auditwheel
pipx >(setup:717): pipx version is 0.16.3
pipx >(setup:718): Default python interpreter is '/opt/_internal/tools/bin/python'
pipx >(package_name_from_spec:323): Determined package name: auditwheel
pipx >(package_name_from_spec:324): Package name determined in 0.0s
creating virtual environment...
pipx >(run_subprocess:135): running /opt/_internal/tools/bin/python -m venv --without-pip /root/.local/pipx/venvs/auditwheel
pipx >(run_subprocess:135): running /root/.local/pipx/venvs/auditwheel/bin/python -c import sysconfig; print(sysconfig.get_path('purelib'))
pipx >(run_subprocess:135): running /root/.local/pipx/shared/bin/python -c import sysconfig; print(sysconfig.get_path('purelib'))
pipx >(run_subprocess:135): running /root/.local/pipx/venvs/auditwheel/bin/python --version
pipx >(_parsed_package_to_package_or_url:128): cleaned package spec: auditwheel
installing auditwheel...
pipx >(run_subprocess:135): running /root/.local/pipx/venvs/auditwheel/bin/python -m pip install --requirement=constraints.txt auditwheel
pipx >(subprocess_post_check_handle_pip_error:297): '/root/.local/pipx/venvs/auditwheel/bin/python -m pip install --requirement=constraints.txt auditwheel' failed
pipx >(subprocess_post_check_handle_pip_error:314): Fatal error from pip prevented installation. Full pip output in file:
    /root/.local/pipx/logs/cmd_2021-07-04_13.02.54_pip_errors.log

pipx >(analyze_pip_output:272): pip seemed to fail to build package:
    auditwheel

pipx >(rmdir:43): removing directory /root/.local/pipx/venvs/auditwheel
Error installing auditwheel.

[root@44456984b498 ~]# cat /root/.local/pipx/logs/cmd_2021-07-04_13.02.54_pip_errors.log
PIP STDOUT
----------
Collecting auditwheel

PIP STDERR
----------
ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
    auditwheel from https://files.pythonhosted.org/packages/90/e1/4af4cbdef79dde083498ae408d629ce5d16728177b970c92344f433257a0/auditwheel-4.0.0-py3-none-any.whl#sha256=96927695ddf27b4edb67291e326908d64ffe272b8a42b9504f283e7ae5ebbc14

[cs01]

“—spec” is the way to do that

[mayeut]

Just a guess, but have you tried passing ‘—spec auditwheel==4.0.0 auditwheel’ to pipx?
I am wondering if this is the issue (it’s not pinned with ==):
pip install --constraint=constraints.txt auditwheel

Adding constraints using --spec to the command-line would work but defeats the purpose of the automation I'm trying to setup (or overly complicates things)

Let's say I have N tools, a requirement.in with those N tools defined, use pip-tools to generate a single constraint file (I'd like to keep it that way - single file - unless there's some dependency issue to keep things simple)

I could in theory just do (pseudo code):

for each tool in requirement.in
    pipx install --pip-args=--constraint=constraints.txt $tool

“—spec” is the way to do that

spec doesn't allow to specify the version for dependencies does it ?

[cs01]

Adding constraints to the command-line would work but defeats the purpose of the automation I'm trying to setup (or overly complicates things)

Agreed that the pipx command is getting a little unwieldy at this point. You could script your tool to generate the pinned version and spec argument to the pipx command.

What would your ideal api to pipx look like in this case? Automatically omit the package name if the requirements argument is passed? Or an additional pipx arg that says not to install the package/spec?

spec doesn't allow to specify the version for dependencies does it ?

it doesn’t, but if the requirements file is passed I don’t think it needs to be. I think pip’s issue is that it’s getting an additional unpinned argument from pipx.

[mayeut]

@cs01,

IMHO, there's nothing to be done in pipx itself (or it would be an ugly workaround). Just reporting the issue here if someone hits it through pipx rather than pip.
I myself searched for constraint in the pypa/pipx repo to check if there was a specific option or problems reported here before trying to do this on my own and thus reported this issue

All will go well once pip is fixed so I would consider this issue to be a placeholder to track progress on pip issue resolution but
at pipx level.

[cs01]

Makes sense. So did the —spec argument work in the mean time?

[mayeut]

Looks like pypa/pip#9020 (comment)

@uranusjr, I saw this one but it's closed and pypa/pip#9243 seems to be exactly what I'm hitting.

So did the —spec argument work in the mean time?

The current workaround is to not install through pipx for now. It's Ieft over code in manylinux with a single pip install -r constraint.txt invocation and multiple ln -s to make binaries available in the PATH. This code predates the inclusion of pipx in manylinux images and I though this would simplify things to use pipx (avoid errors with ln -s, allow easy upgrade via pipx upgrade ... for users, ...).
This will wait for the resolution of the pip issue.

[uranusjr]

Adding constraints to the command-line would work but defeats the purpose of the automation I'm trying to setup (or overly complicates things)

Sorry I was probably too brief. I was trying to suggest using -r and the constraints file together. This is the intended usage described in pypa/pip#9243. So if #88 is implemented we can have a more straightforward way to install packages with a constraints file.

[mayeut]

Adding constraints to the command-line would work but defeats the purpose of the automation I'm trying to setup (or overly complicates things)

Sorry I was probably too brief. I was trying to suggest using -r and the constraints file together. This is the intended usage described in pypa/pip#9243. So if #88 is implemented we can have a more straightforward way to install packages with a constraints file.

@uranusjr,

here are the tests I did that do not work:

for TOOL in ${TOOLS}; do
    pipx install --verbose --pip-args="-c constraints.txt" ${TOOL}
done

# in order to also pin the shared environment
export PIP_CONSTRAINT="constraints.txt"
for TOOL in ${TOOLS}; do
    pipx install --verbose ${TOOL}
done
unset PIP_CONSTRAINT

for TOOL in ${TOOLS}; do
    pipx install --verbose --pip-args="-r constraints.txt" ${TOOL}
done

for TOOL in ${TOOLS}; do
    pipx install --verbose --pip-args="-c constraints.txt -r constraints.txt" ${TOOL}
done

mkdir pipx_install_tmp
pushd pipx_install_tmp

# Install tools
for TOOL in ${TOOLS}; do
    echo "${TOOL}" > ${TOOL}
    pipx install --verbose --pip-args="-c ../constraints.txt -r" ${TOOL}
done

popd
rm -rf pipx_install_tmp

This last one is the same issue as pypa/pip#9243

This is what's working:

mkdir pipx_install_tmp
pushd pipx_install_tmp

# Install tools
for TOOL in ${TOOLS}; do
    ln -s ../constraints.txt ${TOOL}
    pipx install --verbose --pip-args="-c ../constraints.txt -r" ${TOOL}
done

popd
rm -rf pipx_install_tmp

mkdir pipx_install_tmp
pushd pipx_install_tmp

# Install tools
for TOOL in ${TOOLS}; do
    ln -s ../constraints.txt ${TOOL}
    pipx install --verbose --pip-args="-r" ${TOOL}
done

popd
rm -rf pipx_install_tmp

Those install much more packages than required in each venv. Each venv has the same packages but allows for indivudal upgrade/management. The size is not an issue for my use case where I hardlink everything I can afterwards. The time it takes is also contained for now because the tool list is small. It does not allow to pin the shared venv.