setuptools, pip and wheel always in latest version? #2364
Comments
|
I don’t think Pipenv did anything here, and I even filed #1884 specifically because it’s not very convenient to upgrade them. IIRC there is some code to explicitly prevent Pipenv from touching them during installation. Maybe you can try |
uranusjr
added
the
Status: Needs More Information
|
@uranusjr Thanks for the reply! I added the .lock file (must have overread that part, as I only added the Pipfile). As you can see there, setuptools does not get locked. Neither does wheel. Force installing a different version after the venv got created indeed works as a workaround, but it would be much saner to allow version pinning with pipenv in the Pipfile for these packages, too. |
|
Like most packaging tools, pipenv depends upon you having pip and setuptools. As a consequence it also doesn’t allow you to alter which versions of these things you have in your environment using pipenv itself. If pipenv were able to touch these or allowed you to do that, it would be very easy to destroy your environment. I appreciate that in this specific use case of having an outdated artifactory in a distributed system where you don’t want to tell people to run a manual step, it would be more convenient, but in most other cases the behavior as designed is desirable. |
|
@techalchemy well, I just require setuptools to be in a different version. This is not the same as it not being around at all. |
|
@techalchemy also, this is not about setting my global version of setuptools, but the setuptools version, that will be installed into the virtual environment I'm setting up with pipenv. A sane default is of course to choose the latest version of those base tools, but if the user specifically defines a version for them, this should not be ignored! |
|
I agree we need to protect certain core packages, but I would tend to agree with @dvzrv here. We should assume the user knows what (s)he’s doing if those packages are explicitly listed in Pipfile. |
|
Anything that allows you to alter setuptools can break pipenv fundamentally. I see no reason to build in support for our tools to break themselves.
What you’re asking for is for us to build in support for dependency resolution that permits the resolver to return values that could break our own tools. We aren’t going to do that — if everyone internally is pointed at artifactory and they have to use old setuptools versions your company should consider not making the new version available. |
|
I guess the best thing we can do here is to explicitly state what we do and why that is the case. We can emit explicit messages during locking (either strict locking failure or a warning message) to tell the user we can’t obey those specifications because Pipenv needs them to work. |
I beg to differ: There is quite a clear automatic update happening here. As to why we need to achieve this, is that artifactory is probably lacking the speed to deal with newer metadata and twine already making use of it, when using current versions of setuptools. While I agree, that's something on artifactory's side to fix (will start annoying them on monday), it's also awkward to not be able to set a version for the tools you're going to use in the venv and them being force updated to the latest version automatically. |
|
@dvzrv the presence of an upgraded package doesn't really mean anything about pipenv, if you install something else that explicitly and strictly pins a setuptools version it will upgrade setuptools in your virtualenv. If you find evidence of pipenv engaging in this behavior please feel free to provide it, but simply showing state doesn't accomplish that. |
|
@techalchemy I'm not sure what you're trying to achieve here. I've provided the above Pipfile and Pipfile.lock. If I remove wheel and setuptools from them (which are just silently ignored anyways), and do a fresh
I'm unsure what else to provide to prove my point here. All logs and literally everything I provided is "state". If that's not valid, no bug reports are ever valid... That being said: It's not constructive to disregard the collection of information by a participant to reproduce an environment, in which things do not work as intended, while at the same time not providing prove, that this is indeed intended or even documented behavior or disregarding concerns as to why a certain behavior should be changed. This only feels condescending. I would really like to be able to assume, that with pipenv one can create reproducible environments for projects. With the current example I wonder if this can really be achieved if there's no control over certain packages and the behavior is not clearly being defined regarding those I have no control over. |
|
You misunderstand. This isn’t something pipenv does. Obviously I cannot prove a negative so you will have to prove that absent your other requirements, in a brand new environment, pipenv is upgrading setuptools. Since you are filing a bug report against pipenv you need to show that there is a bug — I don’t have your system or your environment, and while I appreciate that this is frustrating, it is not reasonable to ask us to chase down issues which we can’t reproduce and for which the only evidence is that after doing many things, an undesirable state is attained. Surely you can understand why I am pushing back rather hard. The tenor of this issue has been that an outcome occurred and absent any other lines of inquiry and with resistance to the idea that there could possibly be any other cause, the conclusion you came to is that we are responsible for investigating and solving this problem — and that is a bit antagonistic. I get your frustration. However, we are a small team of volunteers with limited resources. If you find a bug you are welcome to report it, but unless you provide sufficient information to reproduce it and evidence that it is a problem in pipenv, we probably aren’t going to spend any resources on it. If the bug is important to you, we expect that you will at least assist us in determining whether it exists rather than assuming it does. This project is a collaborative open source one — basically the process works better if you collaborate instead of blame. To your last point. I already described the behavior. Go ahead and create a fresh environment by just running |
|
@dvzrv Allow me to explian. @techalchemy was trying to tell you that Pipenv does not actively enforce setuptools, pip, and wheel in the virtualenv it manages. What you provided were indirect evidence, e.g. “no-one else enforece them, so it must be Pipenv that did”. This can, however, be disproved by direct evidence, i.e. the source code of Pipenv, and as maintainers of this project, we can tell you with reasonable certainty that Pipenv does not pin setuptools to 39.2.0. Direct evidence trumps directict evidence, and in general can only be disproved by direct evidence—i.e. you need to find exactly where and how Pipenv pins setuptools. To draw a bad analogy—my neighbour accouses my dog for stealing things from his garden yesterday. I don’t think she did. How can I prove it? I can’t, since I would have nothing to give if she didn’t take anything to begin with. My neighbour would either need to catch him doing it, or find the thing belong to him in my house. It doesn’t help if he said “well my dog didn’t take it, it must be yours”. |
I'm unsure where you read that into any of my statements, but I can assure you, that there was no passive-aggressive intent. If you feel so, I apologize.
I'm fully aware of that and I also don't appreciate a passive-aggressive tone. I'm trying to provide you with information. I'm not blaming you. I'm stating what I see as output of the process I'm describing. Let's rewind please, to help me understand. I'll use your example (
This is what I would have to prove wrong, correct? 1st case: macOS (sierra 10.12.6), with a brew-installed python3, brew installed pipenv. After plainly upgrading In both cases, the only "global" or "system-wide" version of setuptools is This seems to be the same on Windows 10, but I can provide information on the system only early next week. I could try to reproduce this on Linux as well, but my distribution has the latest setuptools shipped and I will need to setup pyenv first, to have an older release of setuptools be the only thing seen. What else would you need as information on the system at hand (macOS)?
@uranusjr How can the observed behavior be explained otherwise? I'm also not saying pipenv is pinning it, but it seems as if it is upgrading those packages to their latest version, although they are available in a specific version on the system (which would be contradictory to @techalchemy's earlier statement). Additionally, I couldn't find specific pinning in pip, or wheel (and also tried with having |
|
Hmmmm... so I’m pretty sure we just spit out calls to |
|
The current version on the Mac system is 16.0.0. I assume it will be the same on Windows (will check tomorrow). |
|
Hello, Great tool, first time reporter here and long time heavy user of many other package managers (npm/yarn, nuget/paket, bower, conan, gem, old-style-pip, cargo and probably a few others, but more casually). As I understand properly, the virtualenv creation (something like : Those three packages (an maybe others) are exceptions and we need to handle them carefully. The behavior I would expect from
Thanks again for creating and maintaining this tool! - Frank |
|
What Frank said ;-) I'd probably be most happy with 2.) but, if that will turn out to be too much of a support-nightmare 1.) (i.e. #2365) is fine, too. In any case, it would be good to have documentation on it, so people like me don't fall under the false assumption of being able to change the versions of those tools using pipenv (with a Pipfile). @uranusjr @techalchemy thanks for looking into it! |
|
I do just want to confirm because I am genuinely surprised -- does making a new virtualenv actually upgrade setuptools and such? I guess you might not be able to confirm this... I will get to the bottom of it though |
|
I'm not quite sure what happens, or how the package version of those exception-packages are set; but here are the following commands I am running in an empty folder with pipenv : and the same commands with a naked virtualenv. The virtual environment seems to use diferent Thanks again for looking into it! |
|
I may be missing things, but different versions of virtualenv can install different versions of pip, setuptools, and wheel (because it vendors them). You need to make sure you’re using the same installation of virtualenv to make a good comparison. |
|
I have virtualenv 15.2.0 |
|
@uranusjr @techalchemy Any progress on this issue? Do you need any additional information? What is your opinion on the following solution : That way Thanks!
|
|
Same issue here. Trying to get Sphinx working, but when setuptools is below a certain version, it throws some errors. In the system wide python, there's a setuptools version of 40.0 installed, but the venv pipenv creates contains an older version, incompatible with sphinx, throwing a So... back to shell scripts to wrap around pipenv... |
|
A direct effect of this behavior can be found in #2924. |
|
Yes, we suffer a lot from pipenv breakages, and I believe it's this issue which is causing unnecessary problems. Every time someone recreates their pipenv, or a CI build runs with new dependencies, and there's a pip upgrade, then IMO, each version of pipenv should track which versions of |
|
Not being able to lock the version of pip that is installed inside the VENV and used for installing packages is a BIG issue. I have a pipeline that is currently stuck because latest version of pip has introduced a bug in it (pypa/pip#6163), and pipenv does not let me specify which version of pip to use, uses latest and breaks my build. How are you supposed to achieve reproducible builds if you cannot have consistent build tools version ? |
|
For reference, based on @uranusjr 's comment: |
|
Hello |
* Remove setuptools from pipfile as it is ignored by pipenv. Also see open issue at pypa/pipenv#2364 * Removed pinned versions of build tools, we aim to keep it up-to-date with latest. * Keeping arcgis on 1.6, cannot install newer versions on RHEL7 build agents. Not critical here as arcgis is only used for unit tests.
|
Hi, because of this duplicate - #4149 I wanted to ask if there is any update on this matter? |
|
This has suddenly become a huge problem for me. My company has a legacy python 2.7 app that is very dated and I have no intention of trying to overhaul it or update its dependencies or anything like that. A previous engineer moved our build system to use pipenv about two years ago, so that we could just lock the dependencies and rebuild the virtual env anytime using He didn't realize, though, that even without updating pipenv itself, the Of course, this kind of stuff is only going to continue to get worse for us as python 2.7 is dead, and pip will understandably be dropping support for it entirely. So I imagine that pretty much all of these post install scripts will eventually break-- again, without even updating pipenv itself or anything else on the system-- unless we can pin the pip version these builds are using. Again, I get the idea that we should get with the times and stop relying on all this old stuff, but we kind of don't have a choice right now. We need to do maintenance on this thing until I can convince my company to let us kill it forever, but because we can't do this one simple thing, I can no longer deploy it at all. It will likely to force me to find a way to either commit our .venv, or somehow find a way to consistently rebuild it using vanilla pip. On top of that, I'm really puzzled as to how this couldn't end up being a problem inside if pipenv itself. I can't imagine that pipenv doesn't make some assumptions about how pip works that could break pipenv as pip is updated. People might expect the occasional problem when updating their build tools, but this can happen without updating pipenv at all. As is happening with us. :\ |
|
I've encountered this issue recently and here is what I've found. This is on Ubuntu 20.04 and I don't know if things are similar on macOS / Windows. From the package description of pipenv sync || true # expected to fail because of incompatible setuptools
pipenv run python -m pip install 'setuptools~=45.2.0' # the version I need
pipenv sync # this time it will workHope this helps. Update: Another way to do this is |
|
I just stumbled across this today. pipenv is 2022.8.17 ships with setuptools 63.2.0 which it uses to initialize venvs. However, it never upgrades setuptools due to: Lines 61 to 67 in 697f1a9 Lines 732 to 736 in 697f1a9 This wouldn't be an much of issue for most folks due to the above code. venvs would use setuptools that ships with pipenv and that's it. However, for us it's problematic as we use some custom (much faster) logic instead of So I think if pipenv folks really do not want to update Here you can see, the empty However, from scanning this bug it looks like folks would really like to use newer pip, setuptools, wheel packages so perhaps, given the guidance is to install pipenv through pipx, |
|
@stewartmiles This is due to a recent change to exclude |
|
@matteius there are a huge number of issues in this repo so I felt bad adding to the list and did my best to find the most relevant issue. Thanks for letting me know this is a recent issue. w.r.t the failing test, I took a look at the Ubuntu logs at https:/pypa/pipenv/runs/7908654820?check_suite_focus=true It's interesting that pip depends upon distutils which has now been vendored into setuptools and should soon be gone (I stumbled across pypa/pip#10647). I'm confused about how pipenv is tickling this though since pip needs to load distutils modules to install or uninstall and therefore they should be resident in memory as execution continues. Using pip directly the setuptools upgrade works fine: When using I think the stack trace here provides a clue to what's going on. The stack is pointing at the pip install in my pipx managed venv for pipenv (i.e /home/stewart/.local/pipx/venvs/pipenv) not the one in the newly created in the temporary directory (i.e /tmp/tmp.9ZqA1EKQ0E/.venv). So we've got a case of crossing the venvs, i.e we're trying use python in the pipenv managed venv (under /tmp/tmp.9ZqA1EKQ0E/.venv) where pip is installed in a separate venv (under /home/stewart/.local/pipx/venvs/pipenv). Changing: https:/pypa/pipenv/blob/main/pipenv/core.py#L1528-L1532 to: pip_command = [
project_python(project, system=allow_global),
"-m", "pip",
"install",
]runs pip installed in the venv. Using this code we end up with something that works: So this leaves me with questions:
|
@stewartmiles There are a few patches we support of pip, include some for resolver specifiers, and also an important one is for index restricted packages to prevent package confusion attacks by requiring that every package required calls out a specific index, or uses only the default.
There are a few reasons. Pipenv has had a split between using the vendor'd pip and the user pip for a while and it created its own array of problems. One is because there have been at least a handful or more issues that related to inconsistent behaviors between different versions of pip the user might have installed and what pipenv expects. Plus we rely heavily on pip internals and already vendor in the latest version of pip or typically aren't far behind it, so we can have consistency in what is expected to be executed, and the same patches to pip would apply. I've seen some other issue reports gets closed out as a result of this change, and it allowed us to not explicitly require pip be installed to have pipenv installed. I would like to figure out how to resolve this without reverting back to requiring a user/system pip and re-igniting those other types of issues. I think there are other pipenv bugs related to which python is utilized, so maybe the answer lies in Also odd is that python 3.10 doesn't have this issue, but < 3.10 struggles on this setuptools upgrade. |
|
@stewartmiles thanks to your comment I now have the PR passing for allowing installation of the |
|
This is fixed in |
|
@matteius great news that you're able to get this working with my minimal example. I'll give it a try. Hopefully you noticed that my patch ended up using pip in the venv - which is the pipenv patched pip - not the system pip. Since pip can be upgraded I guess this is brittle w.r.t pipenv's pip internal usage. Though using the patched pip is brittle as pip such an active project, it makes me wonder whether pipenv maintainers have considered / tried to upstream pip patches rather than carrying them across each release? |
|
@stewartmiles -- I cannot say that I agree that using the vendor'd patched pip is brittle -- it creates runtime consistency across users of pipenv and reduces a subset of issues that I have seen. I would argue that trying to rely on the system pip has its own brittleness to it, especially by where pipenv imports pip internals which pretty much means it has always had a vendor'd version of
Yes but upstream is complicated by the fact we use pip internals and the pip maintainers are not interested in patches that affect pip internals without having a direct use case by the pip CLI. For more insight review the discussion on the PR where I tried to patch pip internals for the index restrictions, a patch that pipenv actively maintains: pypa/pip#10964 The set of patches is minimal, documented and maintained as patch files in pipenv: https:/pypa/pipenv/tree/main/tasks/vendoring/patches/patched |
|
@matteius oh I agree with vendoring pip into pipenv, that makes a heap of sense to me. You install pipenv and venvs are seeded with a consistent version of pip that works consistently without dealing with various patched versions of pip that are shipped by OS vendors. The main thing that worried me a bit was using pip embedded in pipenv to manage a venv rather than using the version of pip - chosen by the user - within the venv. I guess if you have a set of patches that apply consistently without much maintenance, such that each pipenv release can integrate a new version of pip and pipenv doesn't upgrade pip in the venv (since users run pip without pipenv to test things out), I guess that's just fine. I had a glance at the patches and I'm surprised some of these can't be upstreamed e.g in https:/pypa/pipenv/blob/main/tasks/vendoring/patches/patched/pip22.patch :
Anyway, I get where you're coming from OSS is definitely hard 😉 Thanks again! |
With current pipenv (
2018.5.18, installed with brew, but same behavior also observed on Windows), I'm unable to version-pin setuptools, pip or wheel. However, the circumstances require it to be version pinned, as other tools (twine) will otherwise try to upload a metainfo version they can not cope with to an artifactory instance. Installing setuptools and wheel in a specific version globally/per-user fixes this, but that's also not a solution for our build environment.(btw. the version scheme is still wrong in the
--versionoutput! :-)).Why are these packages excluded from version pinning and can and/or will this be changed?
Expected result
Be able to install setuptools, wheel and pip in pinned versions.
Actual result
setuptools, pip, wheel get installed in their latest version, no matter what.
Steps to replicate
Pipfile
Pipfile.lock
pipenv installpipenv run pip3 listThe text was updated successfully, but these errors were encountered: