Explore converting pip (or parts of it) into a zipped package

[pradyunsg]

Pulled out of #10462

Where would this be useful

Right now, we have at least two cases where this would be useful:

• The pip package, in its entirety, is packaged into a zip file and made available to the isolated build environments, to ensure that we're running the exact same copy in the PEP 517 isolated build environment.
• There are a substantial number of files that get copied when creating a venv with pip, which serves to slow down the creation process of the venv. Switching to a zip'd package would help with speeding that up; albeit at some runtime costs.

What we could do

There's multiple options:

• A single giant bundle, that is effectively a zipapp.
• A couple of bundles, one for all our vendored packages and one for our _internal stuff.
• A bundle for vendored stuff, which would effectively need a redo of our vendoring setup.
• more?

Why should we not do it?

IDK, this sounds like it will have a lot of churn? Complexity? Potential slowdown of wheel-based installs?

[DiddiLeija]

I'm +1 on zipping our vendored libraries. Even when this would require a rework of many parts of our code, this will make Pip lighter, and then, faster.

However, I don't like a lot the idea of compressing the _internal part. It is big, but I don't know if we should move it.

[uranusjr]

Note that certifi does not work in a zip bundle prior to 3.7 (it uses importlib.resources for zip support), so options other from 3 are not viable before 2022. And after that, we'll need to fix the temporary file check in tests, which would break with importlib.resources.get_path() as mentioned in #9689.

If all the roadblocks are cleared, the best option would be to bundle the entire pip package as one bundle, since that benefits isolated environment setup the most (the overhead would become one single file copy operation).

[pradyunsg]

I mean... we can probably exclude certifi from our bundling shenenigans, and... that basically opens up from all the options. :P

[pfmoore]

My intention when I made the comment was just to zip our vendored libraries, as you say changing our vendoring mechanism. If we use zip -9, the size of our (current) vendored code goes down from 4.9M to 1.4M. This is quite a lot. I'd assumed we could simply zip whatever we could, leave anything difficult (e.g. certifi) unzipped, and then just add _vendor/vendored.zip (assuming that's where we put the zipfile) to sys.path in our startup code.

I'm confused by how the "zip everything" option would work. What would the pip wheel contain, and what would the startup code look like in that case? (And while I know we don't support such usage, how would people like pip-tools access the pip internals if we did this? I don't want to gratuitously break them for something this minor). I'm having trouble picturing what this option would look like in practice, and how it would differ from a "standard" zipapp that doesn't even get installed, like virtualenv provides.

we can probably exclude certifi from our bundling shenenigans

We could also do what we do in get-pip.py and extract the certificate bundle to a temp file.

[pradyunsg]

Closing this out, since we now have zipapp discussions happening and the zip-creation-for-running-in-environment problem is resolved.