context.prefetchInputs is slow on large set of inputs
#20555
Assignees
Labels
P1
I'll work on this now. (Assignee required)
team-Remote-Exec
Issues and PRs for the Execution (Remote) team
type: bug
Comments
coeuvre
added
the
team-Remote-Exec
coeuvre
changed the title
context.prefetchInputs is slow on large inputscontext.prefetchInputs is slow on large set of inputs
|
The repro used for the screenshot in #8230 (comment) involves remote cache. The downloads may contribute to large portion of When trying the repro locally with a clean disk cache, I only see ~400ms span of |
coeuvre
added
P1
coeuvre
added a commit
to coeuvre/bazel
that referenced
this issue
Dec 15, 2023
Use a pre-allocated array to hold the intermediate transfers to avoid allocations. Replace some of RxJava code with Futures to avoid RxJava overheads. This improves the perfromance of prefetchInputs on a large set of inputs from ~400ms to ~16ms. Fixes bazelbuild#20555.
bazel-io
pushed a commit
to bazel-io/bazel
that referenced
this issue
Jan 2, 2024
Use a pre-allocated array to hold the intermediate transfers to avoid allocations. Replace some of RxJava code with Futures to avoid RxJava overheads. This improves the perfromance of prefetchInputs on a large set of inputs from ~400ms to ~16ms. Fixes bazelbuild#20555. Closes bazelbuild#20557. PiperOrigin-RevId: 595226013 Change-Id: If5296fa6b3c0166b95cfca4281255e523724a41f
github-merge-queue bot
pushed a commit
that referenced
this issue
Jan 8, 2024
Use a pre-allocated array to hold the intermediate transfers to avoid allocations. Replace some of RxJava code with Futures to avoid RxJava overheads. This improves the perfromance of prefetchInputs on a large set of inputs from ~400ms to ~16ms. Fixes #20555. Closes #20557. Commit 915fb3e PiperOrigin-RevId: 595226013 Change-Id: If5296fa6b3c0166b95cfca4281255e523724a41f Co-authored-by: Chi Wang <[email protected]> Co-authored-by: Xùdōng Yáng <[email protected]> Co-authored-by: Ian (Hee) Cha <[email protected]> Co-authored-by: Yun Peng <[email protected]>
copybara-service bot
pushed a commit
that referenced
this issue
Jan 17, 2024
*** Reason for rollback *** Might cause build to hang forever. b/320630578 *** Original change description *** Optimize prefetchInputs. Use a pre-allocated array to hold the intermediate transfers to avoid allocations. Replace some of RxJava code with Futures to avoid RxJava overheads. This improves the perfromance of prefetchInputs on a large set of inputs from ~400ms to ~16ms. Fixes #20555. Closes #20557. PiperOrigin-RevId: 599135847 Change-Id: Idae6a1c57e634d16091e31e097b16ca97a67e62d
fweikert
pushed a commit
that referenced
this issue
Jan 25, 2024
*** Reason for rollback *** Might cause build to hang forever. b/320630578 *** Original change description *** Optimize prefetchInputs. Use a pre-allocated array to hold the intermediate transfers to avoid allocations. Replace some of RxJava code with Futures to avoid RxJava overheads. This improves the perfromance of prefetchInputs on a large set of inputs from ~400ms to ~16ms. Fixes #20555. Closes #20557. PiperOrigin-RevId: 599135847 Change-Id: Idae6a1c57e634d16091e31e097b16ca97a67e62d
copybara-service bot
pushed a commit
that referenced
this issue
Jan 30, 2024
Use a pre-allocated array to hold the intermediate transfers to avoid allocations. Replace some of RxJava code with Futures to avoid RxJava overheads. This improves the perfromance of prefetchInputs on a large set of inputs from ~400ms to ~16ms. Fixes #20555. Closes #20557. Automated rollback of commit 162cacd. *** Reason for rollback *** unknown commit is the root cause, not this one. *** Original change description *** Automated rollback of commit 915fb3e. *** Reason for rollback *** Might cause build to hang forever. b/320630578 *** Original change description *** Optimize prefetchInputs. *** PiperOrigin-RevId: 602680109 Change-Id: I357dadd499fecf295e8e87c6fe2892578f3109b1
fweikert
pushed a commit
that referenced
this issue
Feb 5, 2024
*** Reason for rollback *** Might cause build to hang forever. b/320630578 *** Original change description *** Optimize prefetchInputs. Use a pre-allocated array to hold the intermediate transfers to avoid allocations. Replace some of RxJava code with Futures to avoid RxJava overheads. This improves the perfromance of prefetchInputs on a large set of inputs from ~400ms to ~16ms. Fixes #20555. Closes #20557. PiperOrigin-RevId: 599135847 Change-Id: Idae6a1c57e634d16091e31e097b16ca97a67e62d
|
A fix for this issue has been included in Bazel 7.1.0 RC1. Please test out the release candidate and report any issues as soon as possible. Thanks! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Description of the bug:
As shown in #8230 (comment),
context.prefetchInputstakes a couple of seconds on large set of inputs even if all of the inputs are already on the disk. For a no-op prefetch, it should be fast.Which category does this issue belong to?
No response
What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.
No response
Which operating system are you running Bazel on?
No response
What is the output of
bazel info release?No response
If
bazel info releasereturnsdevelopment versionor(@non-git), tell us how you built Bazel.No response
What's the output of
git remote get-url origin; git rev-parse master; git rev-parse HEAD?No response
Is this a regression? If yes, please try to identify the Bazel commit where the bug was introduced.
No response
Have you found anything relevant by searching the web?
No response
Any other information, logs, or outputs that you want to share?
No response
The text was updated successfully, but these errors were encountered: