RECORD size and hash do not reflect rewritten shebangs

[LukeShu]

Description

When installing a wheel, pip rewrites the shebang in scripts in {dist}-{ver}.dist-info/scripts/ from #!python to the appropriate path to the Python interpreter. However, the hash and size in RECORD for the script correspond to the original version of the script, not the rewritten version.

Expected behavior

I expected the RECORD to reflect the files that actually get installed, so that an install can be integrity-checked.

pip version

20.3.4

Python version

3.9.9

OS

Parabola GNU/Linux-libre (like Arch Linux)

How to Reproduce

1. Use pip to install a wheel that contains a {dist}-{ver}.dist-info/scripts/* script with a #!python shebang. One such wheel is websocket-client 0.57.0.
2. Check whether the hash and size of the script match what got recorded in RECORD.

Output

$ wget https://files.pythonhosted.org/packages/4c/5f/f61b420143ed1c8dc69f9eaec5ff1ac36109d52c80de49d66e0c36c3dfdf/websocket_client-0.57.0-py2.py3-none-any.whl
…
$ pip install --ignore-installed --no-deps --prefix=testdir ./websocket_client-0.57.0-py2.py3-none-any.whl
…

$ # Observe what the resulting RECORD says
$ grep bin/w testdir/lib/python3.9/site-packages/websocket_client-0.57.0.dist-info/RECORD 
../../../bin/wsdump.py,sha256=S54et6zebnxb2VJcgBadSnvXblK1iBF93ap54hlc5O8,6403

$ # Observe whether this matches the resulting script file
$ sha256sum testdir/bin/wsdump.py | xargs python -c 'import sys, base64; print(base64.b64encode(bytes.fromhex(sys.argv[1])).decode("utf-8"))'
6GQkITdeQFmlpL7/T9+O/X0sWsKeddIZCvwtU0ld+hc=
$ wc -c < testdir/bin/wsdump.py 
6412

$ # Observe whether this matches the original script in the wheel
$ bsdtar xfO websocket_client-0.57.0-py2.py3-none-any.whl websocket_client-0.57.0.data/scripts/wsdump.py | sha256sum | xargs python -c 'import sys, base64; print(base64.b64encode(bytes.fromhex(sys.argv[1])).decode("utf-8"))'
S54et6zebnxb2VJcgBadSnvXblK1iBF93ap54hlc5O8=
$ bsdtar xfO websocket_client-0.57.0-py2.py3-none-any.whl websocket_client-0.57.0.data/scripts/wsdump.py |wc -c
6403

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[pradyunsg]

Is this reproducible with the latest pip release?

[LukeShu]

Is this reproducible with the latest pip release?

Whoop, I get used to thinking that Arch/Parabola have the latest everything, but I guess Arch has a blocker for upgrading to pip 21.

OK, so using a virtualenv to get the latest pip: yes, it is reproducible with the latest pip release.

$ python -m virtualenv venv                                                                                                                                                                                                      
created virtual environment CPython3.9.9.final.0-64 in 468ms
  creator CPython3Posix(dest=…/venv, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/lukeshu/.local/share/virtualenv)
    added seed packages: pip==21.3.1, setuptools==59.2.0, wheel==0.37.0
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator

$ . venv/bin/activate

(venv) $ pip --version
pip 21.3.1 from …/venv/lib/python3.9/site-packages/pip (python 3.9)

(venv) $ pip install --ignore-installed --no-deps --prefix=testdir ./websocket_client-0.57.0-py2.py3-none-any.whl
Processing ./websocket_client-0.57.0-py2.py3-none-any.whl
Installing collected packages: websocket-client
Successfully installed websocket-client-0.57.0

(venv) $ grep bin/w testdir/lib/python3.9/site-packages/websocket_client-0.57.0.dist-info/RECORD 
../../../bin/wsdump.py,sha256=S54et6zebnxb2VJcgBadSnvXblK1iBF93ap54hlc5O8,6403

[uranusjr]

I do wonder what changed during the time, there’s nothing I recall around this. Maybe this is not even a pip issue but an Arch issue? Could you try versions between 20.3.4 and 21.3.1 and see if the behaviour changed in a particular version?

[pfmoore]

@uranusjr am I confused here? I thought the OP stated that the behaviour was the same in 20.3.4 and 21.3.1? Which makes this feel like a straight bug. In fact, I just confirmed it with Ubuntu (on WSL) with Python 3.8 and pip 21.3.1.

I suspect this is "simply" a case of us copying RECORD from the wheel without recalculating the hash for scripts where we rewrite the shebang. It's probably not been reported before because (a) scripts are uncommon (most people use entry points these days) and (b) very few people check hashes, I suspect.

[pradyunsg]

We do compute these things for generated files.

https:/pypa/pip/blob/main/src/pip/_internal/operations/install/wheel.py#L266

I think we might want to start validating these hashes from wheels at some point, similar to how we'd want to validate the hashes by default in some contexts and get TUF-based protection of all the index interactions. 🤷🏻‍♂️

[pradyunsg]

And we check for files that have been changed; to recompute hashes and sizes.

https:/pypa/pip/blob/main/src/pip/_internal/operations/install/wheel.py#L260

Someone should step through this chunk of code with a debugger to figure out what is going wrong here; because we certainly have code to account for changed files.