Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update vendored dependencies, relevant API usage, and CI config #4169

Merged
merged 49 commits into from
Apr 12, 2020

Conversation

techalchemy
Copy link
Member

Update vendored dependencies and invocations

  • Update vendored and patched dependencies
    • Update patches on piptools, pip, pip-shims, tomlkit
  • Fix invocations of dependencies
    • Fix custom InstallCommand instantiation
    • Update PackageFinder usage
    • Fix Bool stringify attempts from tomlkit

Updated vendored dependencies:

  • attrs: 18.2.0 => 19.1.0

  • certifi: 2018.10.15 => 2019.3.9

  • cached_property: 1.4.3 => 1.5.1

  • cerberus: 1.2.0 => 1.3.1

  • click: 7.0.0 => 7.1.1

  • click-completion: 0.5.0 => 0.5.1

  • colorama: 0.3.9 => 0.4.3

  • contextlib2: (new) => 0.6.0.post1

  • distlib: 0.2.8 => 0.2.9

  • funcsigs: (new) => 1.0.2

  • importlib_metadata 1.3.0 => 1.5.1

  • importlib-resources: (new) => 1.4.0

  • idna: 2.7 => 2.9

  • jinja2: 2.10.0 => 2.11.1

  • markupsafe: 1.0 => 1.1.1

  • more-itertools: (new) => 5.0.0

  • orderedmultidict: (new) => 1.0

  • packaging: 18.0 => 19.0

  • parse: 1.9.0 => 1.15.0

  • pathlib2: 2.3.2 => 2.3.3

  • pep517: (new) => 0.5.0

  • pexpect: 4.6.0 => 4.8.0

  • pip-shims: 0.2.0 => 0.5.1

  • pipdeptree: 0.13.0 => 0.13.2

  • pyparsing: 2.2.2 => 2.4.6

  • python-dotenv: 0.9.1 => 0.10.2

  • pythonfinder: 1.1.10 => 1.2.2

  • pytoml: (new) => 0.1.20

  • requests: 2.20.1 => 2.23.0

  • requirementslib: 1.3.3 => 1.5.4

  • scandir: 1.9.0 => 1.10.0

  • shellingham: 1.2.7 => 1.3.2

  • six: 1.11.0 => 1.14.0

  • tomlkit: 0.5.2 => 0.5.11

  • urllib3: 1.24 => 1.25.8

  • vistir: 0.3.0 => 0.5.0

  • yaspin: 0.14.0 => 0.14.3

  • zipp: 0.6.0

  • Removed vendored dependency cursor.

@techalchemy techalchemy added Type: Vendored Dependencies This issue affects vendored dependencies within pipenv. Priority: Critical This issue is critical and affects usability or core functionality. Type: Maintenance 🚧 This issue pertains to maintenance of pipenv itself. labels Mar 31, 2020
@techalchemy techalchemy added this to the March 2020 Release milestone Mar 31, 2020
@techalchemy techalchemy force-pushed the feature/vendor-update branch 5 times, most recently from 2af12db to 32cb35b Compare March 31, 2020 21:04
@frostming frostming linked an issue Apr 1, 2020 that may be closed by this pull request
@techalchemy techalchemy force-pushed the feature/vendor-update branch 2 times, most recently from ba16187 to 87d4d8e Compare April 1, 2020 04:23
@brainwane
Copy link
Contributor

btw @davidstaheli do you have any time to help us wrangle Azure pipelines? We're having a problem here https://dev.azure.com/pypa/pipenv/_build/results?buildId=21032&view=logs&j=89390fe7-33d2-53e5-d9b1-9b3b600e8bb2&t=154e6688-01f7-599c-c8c6-29bc92fc1d2f where it's unclear whether the "Run integration tests" step is kicking off at all.

@brainwane
Copy link
Contributor

brainwane commented Apr 1, 2020

Fun fact: Azure jobs time out and sometimes that is confusing.

According to the past-180-days failure report pipenv's builds started failing between 2020-01-10 and 2020-01-12 -- we can't see those builds because they are over 30 days old, but we suspect that, at some point there, some runs went over the 60 minute mark and thus things started failing (example).

So this PR is still in progress as Dan splits up the tests so each one takes under 60 minutes, because that sounds faster than fiddling with the timeout variable and talking to Microsoft support to remind them that -- as a public open source project -- pipenv should get much longer timeouts. (IRC log.) If you want to help, you can join us in IRC, #pypa-dev on Freenode, or comment here with tips.

@techalchemy techalchemy force-pushed the feature/vendor-update branch 8 times, most recently from 2d3b008 to 807bd49 Compare April 5, 2020 23:12
@brainwane
Copy link
Contributor

I just spoke with Dan. Turns out the Windows problem that stymied him for a while was host key validation, which he has now fixed, so Windows tests are now running and actually reporting failures. The current failures require that he normalize capitalization of the directories/files in the test environments, and so do not seem to reflect any actual bugs in pipenv functionality. So that's good news!

Mac and Linux tests are evidently passing. So once he gets those Windows tests passing, he should be able to merge the branch and get that prerelease out (#3369).

@brainwane brainwane changed the title Update vendored dependencies and relevant api usage Update vendored dependencies, relevant API usage, and CI config Apr 6, 2020
@brainwane brainwane mentioned this pull request Apr 9, 2020
@brainwane
Copy link
Contributor

From IRC in the past few days:

windows failures don't appear in azure test results
the builds fail but nothing indicates what went wrong
https://dev.azure.com/pypa/pipenv/_build/results?buildId=21643&view=results why doesn't windows report its failures :|
it fails but reports nothing

Dan has tried runing a subset of the tests in the CI and that didn't help. Azure and Windows testing experts: please help.

@pietrodn
Copy link

pietrodn commented Apr 9, 2020

If you click on "Run integration tests" it is impossible to get to the actual test report. However if you click on the name of the test environment, such as "TestWindows Python38", you are shown a link such as "93,6% tests passed" on the right: if you click on it, you get the actual outcome of the tests.

I'll include an image to be more clear.
image

Here's the failure: log, code

@brainwane
Copy link
Contributor

brainwane commented Apr 9, 2020

@pradyunsg also started poking at this problem in a branch. I don't think he has a lot of time to submit pull requests about it; I link to it in case it serves as a basis and helps anyone else submit pipenv PRs or recommend things for Dan to do in this PR.

techalchemy and others added 21 commits April 10, 2020 12:14
Signed-off-by: Dan Ryan <[email protected]>
- Use ramdisk on windows without subdirectory for tempdir path

Signed-off-by: Dan Ryan <[email protected]>
Signed-off-by: Dan Ryan <[email protected]>
Signed-off-by: Dan Ryan <[email protected]>
- Ditch timeout runner and update test modules

Signed-off-by: Dan Ryan <[email protected]>
- Fix test plugin for pypi runner

Signed-off-by: Dan Ryan <[email protected]>
- only fix egg links in the project environment

Signed-off-by: Dan Ryan <[email protected]>
- Fix azure pipelines script

Signed-off-by: Dan Ryan <[email protected]>
Done for a cleaner diff when moving this block.
This also needs renaming the pipeline on Azure Pipeline's Web UI
@techalchemy techalchemy merged commit ca34b2e into master Apr 12, 2020
@techalchemy techalchemy deleted the feature/vendor-update branch April 12, 2020 02:42
ilai-deutel added a commit to ilai-deutel/PKGBUILDs that referenced this pull request Apr 12, 2020
upgpkg: python-pipenv-git 2018.11.26.r751.g5c01c682-1
fwojciak pushed a commit to fwojciak/pipenv that referenced this pull request May 29, 2020
2020.5.28 (2020-05-28)
======================

Features & Improvements
-----------------------

-   `pipenv install` and `pipenv sync` will no longer attempt to install satisfied dependencies during installation. pypa#3057, pypa#3506
-   Added support for resolution of direct-url dependencies in `setup.py` files to respect `PEP-508` style URL dependencies. pypa#3148
-   Added full support for resolution of all dependency types including direct URLs, zip archives, tarballs, etc.
    -   Improved error handling and formatting.
    -   Introduced improved cross platform stream wrappers for better `stdout` and `stderr` consistency. pypa#3298
-   For consistency with other commands and the `--dev` option description, `pipenv lock --requirements --dev` now emits both default and development dependencies. The new `--dev-only` option requests the previous behaviour (e.g. to generate a `dev-requirements.txt` file). pypa#3316
-   Pipenv will now successfully recursively lock VCS sub-dependencies. pypa#3328
-   Added support for `--verbose` output to `pipenv run`. pypa#3348
-   Pipenv will now discover and resolve the intrinsic dependencies of **all** VCS dependencies, whether they are editable or not, to prevent resolution conflicts. pypa#3368
-   Added a new environment variable, `PIPENV_RESOLVE_VCS`, to toggle dependency resolution off for non-editable VCS, file, and URL based dependencies. pypa#3577
-   Added the ability for Windows users to enable emojis by setting `PIPENV_HIDE_EMOJIS=0`. pypa#3595
-   Allow overriding `PIPENV_INSTALL_TIMEOUT` environment variable (in seconds). pypa#3652
-   Allow overriding `PIP_EXISTS_ACTION` evironment variable (value is passed to pip install). Possible values here: <https://pip.pypa.io/en/stable/reference/pip/#exists-action-option> Useful when you need to `PIP\_EXISTS\_ACTION=i` (ignore existing packages) - great for CI environments, where you need really fast setup. pypa#3738
-   Pipenv will no longer forcibly override `PIP_NO_DEPS` on all vcs and file dependencies as resolution happens on these in a pre-lock step. pypa#3763
-   Improved verbose logging output during `pipenv lock` will now stream output to the console while maintaining a spinner. pypa#3810
-   Added support for automatic python installs via `asdf` and associated `PIPENV_DONT_USE_ASDF` environment variable. pypa#4018
-   Pyenv/asdf can now be used whether or not they are available on PATH. Setting `PYENV_ROOT`/`ASDF_DIR` in a `.env` file allows Pipenv to install an interpreter without any shell customizations, so long as pyenv/asdf is installed. pypa#4245
-   Added `--key` command line parameter for including personal PyUp.io API tokens when running `pipenv check`. pypa#4257

Behavior Changes
----------------

-   Make conservative checks of known exceptions when subprocess returns output, so user won\'t see the whole traceback - just the error. pypa#2553
-   Do not touch Pipfile early and rely on it so that one can do `pipenv sync` without a Pipfile. pypa#3386
-   Re-enable `--help` option for `pipenv run` command. pypa#3844
-   Make sure `pipenv lock -r --pypi-mirror {MIRROR_URL}` will respect the pypi-mirror in requirements output. pypa#4199

Bug Fixes
---------

-   Raise `PipenvUsageError` when \[\[source\]\] does not contain url field. pypa#2373
-   Fixed a bug which caused editable package resolution to sometimes fail with an unhelpful setuptools-related error message. pypa#2722
-   Fixed an issue which caused errors due to reliance on the system utilities `which` and `where` which may not always exist on some
    systems.
-   Fixed a bug which caused periodic failures in python discovery when executables named `python` were not present on the target `$PATH`. pypa#2783
-   Dependency resolution now writes hashes for local and remote files to the lockfile. pypa#3053
-   Fixed a bug which prevented `pipenv graph` from correctly showing all dependencies when running from within `pipenv shell`. pypa#3071
-   Fixed resolution of direct-url dependencies in `setup.py` files to respect `PEP-508` style URL dependencies. pypa#3148
-   Fixed a bug which caused failures in warning reporting when running pipenv inside a virtualenv under some circumstances.
-   Fixed a bug with package discovery when running `pipenv clean`. pypa#3298
-   Quote command arguments with carets (`^`) on Windows to work around unintended shell escapes. pypa#3307
-   Handle alternate names for UTF-8 encoding. pypa#3313
-   Abort pipenv before adding the non-exist package to Pipfile. pypa#3318
-   Don\'t normalize the package name user passes in. pypa#3324
-   Fix a bug where custom virtualenv can not be activated with pipenv shell pypa#3339
-   Fix a bug that `--site-packages` flag is not recognized. pypa#3351
-   Fix a bug where `pipenv --clear` is not working pypa#3353
-   Fix unhashable type error during `$ pipenv install --selective-upgrade` pypa#3384
-   Dependencies with direct `PEP508` compliant VCS URLs specified in their `install_requires` will now be successfully locked during the resolution process. pypa#3396
-   Fixed a keyerror which could occur when locking VCS dependencies in
    some cases. pypa#3404
-   Fixed a bug that `ValidationError` is thrown when some fields are missing in source section. pypa#3427
-   Updated the index names in lock file when source name in Pipfile is changed. pypa#3449
-   Fixed an issue which caused `pipenv install --help` to show duplicate entries for `--pre`. pypa#3479
-   Fix bug causing `[SSL: CERTIFICATE_VERIFY_FAILED]` when Pipfile `[[source]]` has `verify_ssl=false` and url with custom port. pypa#3502
-   Fix `sync --sequential` ignoring `pip install` errors and logs. pypa#3537
-   Fix the issue that lock file can\'t be created when `PIPENV_PIPFILE` is not under working directory. pypa#3584
-   Pipenv will no longer inadvertently set `editable=True` on all vcs dependencies. pypa#3647
-   The `--keep-outdated` argument to `pipenv install` and `pipenv lock` will now drop specifier constraints when encountering editable dependencies.
    -   In addition, `--keep-outdated` will retain specifiers that would otherwise be dropped from any entries that have not been updated. pypa#3656
-   Fixed a bug which sometimes caused pipenv to fail to respect the `--site-packages` flag when passed with `pipenv install`. pypa#3718
-   Normalize the package names to lowercase when comparing used and in-Pipfile packages. pypa#3745
-   `pipenv update --outdated` will now correctly handle comparisons between pre/post-releases and normal releases. pypa#3766
-   Fixed a `KeyError` which could occur when pinning outdated VCS dependencies via `pipenv lock --keep-outdated`. pypa#3768
-   Resolved an issue which caused resolution to fail when encountering poorly formatted `python_version` markers in `setup.py` and `setup.cfg` files. pypa#3786
-   Fix a bug that installation errors are displayed as a list. pypa#3794
-   Update `pythonfinder` to fix a problem that `python.exe` will be mistakenly chosen for virtualenv creation under WSL. pypa#3807
-   Fixed several bugs which could prevent editable VCS dependencies from being installed into target environments, even when reporting
    successful installation. pypa#3809
-   `pipenv check --system` should find the correct Python interpreter when `python` does not exist on the system. pypa#3819
-   Resolve the symlinks when the path is absolute. pypa#3842
-   Pass `--pre` and `--clear` options to `pipenv update --outdated`. pypa#3879
-   Fixed a bug which prevented resolution of direct URL dependencies which have PEP508 style direct url VCS sub-dependencies with
    subdirectories. pypa#3976
-   Honor `PIPENV_SPINNER` environment variable pypa#4045
-   Fixed an issue with `pipenv check` failing due to an invalid API key from `pyup.io`. pypa#4188
-   Fixed a bug which caused versions from VCS dependencies to be included in `Pipfile.lock` inadvertently. pypa#4217
-   Fixed a bug which caused pipenv to search non-existent virtual environments for `pip` when installing using `--system`. pypa#4220
-   `Requires-Python` values specifying constraint versions of python starting from `1.x` will now be parsed successfully. pypa#4226
-   Fix a bug of `pipenv update --outdated` that can\'t print output correctly. pypa#4229
-   Fixed a bug which caused pipenv to prefer source distributions over wheels from `PyPI` during the dependency resolution phase. Fixed an issue which prevented proper build isolation using `pep517` based builders during dependency resolution. pypa#4231
-   Don\'t fallback to system Python when no matching Python version is found. pypa#4232

Vendored Libraries
------------------

- Updated `pip_shims` to support `--outdated` with new pip versions. pypa#3766
- Update vendored dependencies and invocations
  - Update vendored and patched dependencies
  - Update patches on `piptools`, `pip`, `pip-shims`, `tomlkit`
  - Fix invocations of dependencies
  - Fix custom `InstallCommand` instantiation
  - Update `PackageFinder` usage
  - Fix `Bool` stringify attempts from `tomlkit`
  - Updated vendored dependencies:
    -   **attrs**: `18.2.0 => `19.1.0`
    -   **certifi**: `2018.10.15 => `2019.3.9`
    -   **cached\_property**: `1.4.3 => `1.5.1`
    -   **cerberus**: `1.2.0 => `1.3.1`
    -   **click**: `7.0.0 => `7.1.1`
    -   **click-completion**: `0.5.0 => `0.5.1`
    -   **colorama**: `0.3.9 => `0.4.3`
    -   **contextlib2**: `(new) => `0.6.0.post1`
    -   **distlib**: `0.2.8 => `0.2.9`
    -   **funcsigs**: `(new) => `1.0.2`
    -   **importlib\_metadata** `1.3.0 => `1.5.1`
    -   **importlib-resources**: `(new) => `1.4.0`
    -   **idna**: `2.7 => `2.9`
    -   **jinja2**: `2.10.0 => `2.11.1`
    -   **markupsafe**: `1.0 => `1.1.1`
    -   **more-itertools**: `(new) => `5.0.0`
    -   **orderedmultidict**: `(new) => `1.0`
    -   **packaging**: `18.0 => `19.0`
    -   **parse**: `1.9.0 => `1.15.0`
    -   **pathlib2**: `2.3.2 => `2.3.3`
    -   **pep517**: `(new) => `0.5.0`
    -   **pexpect**: `4.6.0 => `4.8.0`
    -   **pip-shims**: `0.2.0 => `0.5.1`
    -   **pipdeptree**: `0.13.0 => `0.13.2`
    -   **pyparsing**: `2.2.2 => `2.4.6`
    -   **python-dotenv**: `0.9.1 => `0.10.2`
    -   **pythonfinder**: `1.1.10 => `1.2.2`
    -   **pytoml**: `(new) => `0.1.20`
    -   **requests**: `2.20.1 => `2.23.0`
    -   **requirementslib**: `1.3.3 => `1.5.4`
    -   **scandir**: `1.9.0 => `1.10.0`
    -   **shellingham**: `1.2.7 => `1.3.2`
    -   **six**: `1.11.0 => `1.14.0`
    -   **tomlkit**: `0.5.2 => `0.5.11`
    -   **urllib3**: `1.24 => `1.25.8`
    -   **vistir**: `0.3.0 => `0.5.0`
    -   **yaspin**: `0.14.0 => `0.14.3`
    -   **zipp**: `0.6.0`
    - Removed vendored dependency **cursor**. pypa#4169

-   Add and update vendored dependencies to accommodate `safety` vendoring:
    -   **safety** `(none)` => `1.8.7`
    -   **dparse** `(none)` => `0.5.0`
    -   **pyyaml** `(none)` => `5.3.1`
    -   **urllib3** `1.25.8` => `1.25.9`
    -   **certifi** `2019.11.28` => `2020.4.5.1`
    -   **pyparsing** `2.4.6` => `2.4.7`
    -   **resolvelib** `0.2.2` => `0.3.0`
    -   **importlib-metadata** `1.5.1` => `1.6.0`
    -   **pip-shims** `0.5.1` => `0.5.2`
    -   **requirementslib** `1.5.5` => `1.5.6` pypa#4188

-   Updated vendored `pip` => `20.0.2` and `pip-tools` => `5.0.0`. pypa#4215
-   Updated vendored dependencies to latest versions for security and bug fixes:
    -   **requirementslib** `1.5.8` => `1.5.9`
    -   **vistir** `0.5.0` => `0.5.1`
    -   **jinja2** `2.11.1` => `2.11.2`
    -   **click** `7.1.1` => `7.1.2`
    -   **dateutil** `(none)` => `2.8.1`
    -   **backports.functools\_lru\_cache** `1.5.0` => `1.6.1`
    -   **enum34** `1.1.6` => `1.1.10`
    -   **toml** `0.10.0` => `0.10.1`
    -   **importlib\_resources** `1.4.0` => `1.5.0` pypa#4226
-   Changed attrs import path in vendored dependencies to always import from `pipenv.vendor`. pypa#4267

Improved Documentation
----------------------

-   Added documenation about variable expansion in `Pipfile` entries. pypa#2317
-   Consolidate all contributing docs in the rst file pypa#3120
-   Update the out-dated manual page. pypa#3246
-   Move CLI docs to its own page. pypa#3346
-   Replace (non-existant) video on docs index.rst with equivalent gif. pypa#3499
-   Clarify wording in Basic Usage example on using double quotes to escape shell redirection pypa#3522
-   Ensure docs show navigation on small-screen devices pypa#3527
-   Added a link to the TOML Spec under General Recommendations & Version Control to clarify how Pipfiles should be written. pypa#3629
-   Updated the documentation with the new `pytest` entrypoint. pypa#3759
-   Fix link to GIF in README.md demonstrating Pipenv\'s usage, and add descriptive alt text. pypa#3911
-   Added a line describing potential issues in fancy extension. pypa#3912
-   Documental description of how Pipfile works and association with Pipenv. pypa#3913
-   Clarify the proper value of `python_version` and `python_full_version`. pypa#3914
-   Write description for `--deploy` extension and few extensions differences. pypa#3915
-   More documentation for `.env` files pypa#4100
-   Updated documentation to point to working links. pypa#4137
-   Replace docs.pipenv.org with pipenv.pypa.io pypa#4167
-   Added functionality to check spelling in documentation and cleaned up existing typographical issues. pypa#4209
archlinux-github pushed a commit to archlinux/aur that referenced this pull request Aug 3, 2022
upgpkg: python-pipenv-git 2018.11.26.r751.g5c01c682-1
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Priority: Critical This issue is critical and affects usability or core functionality. Type: Maintenance 🚧 This issue pertains to maintenance of pipenv itself. Type: Vendored Dependencies This issue affects vendored dependencies within pipenv.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Vendored Jinja2 has known vulnerability
7 participants