Make rez pip useful

[mottosso]

Goal

Make pip great again. Or rather, make pip actually useful.

Motivation

rez pip --install currently creates a Rez package from a Python package on PyPI, which is great. However it also embeds overly specific requirements for the package.

Consider the following example.

$ rez pip --install Qt.py

The resulting package is..

~/packages/Qt.py/1.1.0/platform-windows/arch-AMD64/os-windows-10.0.17763/python-3.6/

Meaning that in order to use this package, you'll need:

• A Windows OS
• Of the exact version 10.0.17763
• Python 3.6
• A 64-bit machine

Even though Qt.py works on..

• Any OS
• Any version of Windows (and Linux, and MacOS)
• Any arch
• Any Python

The same is true for any package, including compiled ones. A compiled PyQt5 would require an exact arch, an exact platform and Python distribution, but not the exact version of OS.

Some packages require Python 2 or 3, but not necessarily the minor version; thus python-3 is sufficient to support all 3.x versions.

Proposal

Ditch rez pip as it exists today, and make this the new default.

$ rez pip --install Qt.py
# Installed using wheels, as per #602

Resulting in:

~/packages/Qt.py/1.1.0/package.py

With options for:

$ rez pip --install Qt.py --variant platform-windows --variant python-3.6

For wheels specific to a particular OS or interpreter, like PySide, it could then look towards a provided --variant python-X to determine which Python to build/download for.

That's assuming you, the one installing a package from pip, knows full well what a package requires and aren't dependent on the information being inferred from its metadata.

However, for when you don't and the information is there:

$ rez pip --install UnfamiliarPackage --auto-variants

Which could look at whatever sources is available to a package on pip to figure out which variants to package it under.

Thoughts?

[j0yu]

We haven't addressed them yet due to our current size and complexity, but those are great points and issues that I definitely support.

There was a point where I noticed it could be an issue e.g. upgrading from even just centos-7.4 to centos-7.6 (testing in docker container first) , none of the pypi packages we have will show up. I have thought about it then that thought went down the rabbit hole of potentially even having (amongst other things):

• auto requirements.txt or pipenv scanning to determine any requirements or variants
• custom package (re)name that's compatible with rez studio's naming conventions e.g. Qt.py released as qt_py

One thing I'll mention, though not sure how relevant it is, is we run our rez pip installs in a Centos container that uses the latest pip and setuptools (installed independently then rez bind) rather than the prehistoric versions that's available via yum. It's just a simple workaround for installing some packages, though I have a strange feeling there maybe some underlying rez workflow requiring rez bind pip and setuptools that's affecting how rez pip behaves when installing packages but never thoroughly looked at the source code or researched the Google groups about it, how rez pip works and any potential implications of other install/dependency problems aside from not being able to handle wheels.

Having worked in bigger companies, I definitely see the benefits your PR and it would help tidy up and prepare our packages for whenever python 3 comes around (now that's another can of delicious worms). Compared with what I'm use to, we're very flexible when it comes to changing our package setups and releases so trying out any improvements are always welcome

[bpabel]

I think the proposal sounds great. A couple other points that I think would be useful.

1. Don't implement rez pip using cmake.

It's a pain to get working on windows, and has a number of dependencies that aren't necessary for doing python installs. Ideally, rez-pip shouldn't require any other dependencies besides what gets installed with rez.

2. Make sure you can generate variants separately.

One issue to look out for whenever you're generating package.py files is to take into account variants that were built in the past from other rez-pip's using a different variant.

With a normal package.py that contains all the variants, you can build/release a single variant and it's fine, because the built/released package.py will list all the variants even if you're only building one.

If you're only generating variants dynamically one-at-a-time, you'll need to look at the existing install/release to get the other variants when generating the package.py file.

[mottosso]

Thank you gentlemen, the implementation for this is currently a PR and has been working well so far (~80 pip packages tested so far); both with explicitly defining variants, auto-detecting them, and installing from other sources like off of disk and straight from GitHub.

Take it for a spin and let me know if you encounter any issues.

• WIP Implements #612, Useful pip #614

[nerdvegas]

In order to be useful, I think rez-pip needs to be pessimistic - ie, it is better to be overly specific wrt requirements than it is to get them wrong, or to depend on the user to know them. The more the user needs to know stuff, the less the'll want to use it. I agree that there needs to be some way to override the requirements yourself however. There is also more to consider beyond what you mention here - we haven't taken varianting into consideration. Ie, should the resultant rez package be varianted on python, or not? That's a more complex topic than I want to cover here as it starts touching on a lot of stuff. A

…

On Wed, Apr 24, 2019 at 7:52 PM Marcus Ottosson ***@***.***> wrote: Goal Make pip great again. Or rather, make pip actually useful. Motivation rez pip --install currently creates a Rez package from a Python package on PyPI, which is great. However it also embeds overly specific requirements for the package. Consider the following example. $ rez pip --install Qt.py The resulting package is.. ~/packages/Qt.py/1.1.0/platform-windows/arch-AMD64/os-windows-10.0.17763/python-3.6/ Meaning that in order to use this package, you'll need: - A Windows OS - Of the exact version 10.0.17763 - Python 3.6 - A 64-bit machine Even though Qt.py works on.. - Any OS - Any version of Windows (and Linux, and MacOS) - Any arch - Any Python The same is true for any package, including compiled ones. A compiled PyQt5 would require an exact arch, an exact platform and Python distribution, but not the exact version of OS. Some packages require Python 2 or 3, but not necessarily the minor version; thus python-3 is sufficient to support all 3.x versions. Proposal Ditch rez pip as it exists today, and make this the new default. $ rez pip --install Qt.py# Installed using wheels, as per #602 Resulting in: ~/packages/Qt.py/1.1.0/package.py With options for: $ rez pip --install Qt.py --require platform-windows --require python-3.6 For wheels specific to a particular OS or interpreter, like PySide, it could then look towards a provided --require python-X to determine which Python to build/download for. That's assuming you, the one installing a package from pip, knows full well what a package requires and aren't dependent on the information being inferred from its metadata. However, for when you don't and the information is there: $ rez pip --install UnfamiliarPackage --auto-require Which could look at whatever sources is available to a package on pip to figure out the requirements needed. Thoughts? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#612>, or mute the thread <https:/notifications/unsubscribe-auth/AAMOUSXB3Q57DYRWJKGWA6TPSAUWZANCNFSM4HICHTQQ> .

[bpabel]

should the resultant rez package be varianted on python, or not?

The current wheel format should tell you this. It has metadata that indicates it it works for both py2/3 or just a specific versions of python.

[mottosso]

There is also more to consider beyond what you mention here - we haven't
taken varianting into consideration

@bpabel is right, have a look at the PR for this, it continues along these lines and takes variants like the ones you mention into account.

• WIP Implements #612, Useful pip #614

[mottosso]

Closing this in favour of #614