-
Notifications
You must be signed in to change notification settings - Fork 2.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Parse less JSON on null builds #6880
Merged
Merged
Commits on May 3, 2019
-
Only execute
exclude_from_backups
onceThis was currently getting executed on all builds, even if the directory already exists. There shouldn't be any reason though to exclude the directory from backups on all builds, and after seeing this get a stack sample in a profile I figured it's best to ensure it only executes once in case the backing system implementation isn't the speediest.
Configuration menu - View commit details
-
Copy full SHA for 8258ca4 - Browse repository at this point
Copy the full SHA 8258ca4View commit details -
Don't allocate in
SourceId::is_default_registry
This gets called quite a lot and doesn't need to allocate in the first place!
Configuration menu - View commit details
-
Copy full SHA for c7e1b68 - Browse repository at this point
Copy the full SHA c7e1b68View commit details -
Parse less JSON on null builds
This commit fixes a performance pathology in Cargo today. Whenever Cargo generates a lock file (which happens on all invocations of `cargo build` for example) Cargo will parse the crates.io index to learn about dependencies. Currently, however, when it parses a crate it parses the JSON blob for every single version of the crate. With a lock file, however, or with incremental builds only one of these lines of JSON is relevant. Measured today Cargo building Cargo parses 3700 JSON dependencies in the registry. This commit implements an optimization that brings down the number of parsed JSON lines in the registry to precisely the right number necessary to build a project. For example Cargo has 150 crates in its lock file, so now it only parses 150 JSON lines (a 20x reduction from 3700). This in turn can greatly improve Cargo's null build time. Cargo building Cargo dropped from 120ms to 60ms on a Linux machine and 400ms to 200ms on a Mac. The commit internally has a lot more details about how this is done but the general idea is to have a cache which is optimized for Cargo to read which is maintained automatically by Cargo. Closes rust-lang#6866
Configuration menu - View commit details
-
Copy full SHA for 6babe72 - Browse repository at this point
Copy the full SHA 6babe72View commit details -
Make registry locking more coarse
This commit updates the locking strategy in Cargo handle the recent addition of creating a cache of the on-disk index also on disk. The goal here is reduce the overhead of locking both cognitively when reading but also performance wise by requiring fewer locks. Previously Cargo had a bunch of fine-grained locks throughout the index and git repositories, but after this commit there's just one global "package cache" lock. This global lock now serves to basically synchronize the entire crate graph resolution step. This shouldn't really take that long unless it's downloading, in which case there's not a ton of benefit to running in parallel anyway. The other intention of this single global lock is to make it much easier on the sources to not worry so much about lock ordering or when to acquire locks, but rather they just assert in their various operations that they're locked. Cargo now has a few coarse-grained locations where locks are held (for example during resolution and during package downloading). These locks are a bit sprinkled about but they have in-code asserts which assert that they're held, so we'll find bugs quickly if any lock isn't held (before a race condition is hit that is)
Configuration menu - View commit details
-
Copy full SHA for 5217280 - Browse repository at this point
Copy the full SHA 5217280View commit details -
Thread through last update time to index cache
Removed in the previous commit this now adds dedicated tracking for the last update.
Configuration menu - View commit details
-
Copy full SHA for 783f22b - Browse repository at this point
Copy the full SHA 783f22bView commit details -
Avoid using mtime information for reusing cache files
Using mtime information is pretty finnicky across platforms, so instead take a different strategy where we embed the sha that a cache file was generated from into the cache file itself. If the registry's sha has changed then we regenerate the cache file, otherwise we can reuse the cache file. This should make cache file generation more robust (any command can generate a cache file to get used at any time) as well as works better across platforms (doesn't run into issues with coarse mtime systems and the like).
Configuration menu - View commit details
-
Copy full SHA for 1daff03 - Browse repository at this point
Copy the full SHA 1daff03View commit details -
Add debug assertions for cache contents
This will largely only get tested during Cargo's own tests, but this commit adds debug assertions where the cache of registry JSON files is always valid and up to date when we consider it being up to date.
Configuration menu - View commit details
-
Copy full SHA for e33881b - Browse repository at this point
Copy the full SHA e33881bView commit details
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.