rustc: document the jobserver #121564
Merged
rustc: document the jobserver #121564
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
rustbot
added
the
S-waiting-on-review
48 tasks
This comment has been minimized.
This comment has been minimized.
To be used to skip the `tab` check in `jobserver.md`. Signed-off-by: Miguel Ojeda <[email protected]>
ojeda
force-pushed
the
rustc-jobserver
branch
from
438470e
to
2834c4c
Compare
rustbot
added
A-testsuite
ehuss
reviewed
Feb 28, 2024
src/doc/rustc/src/jobserver.md
Outdated
|
|
||
| Internally, `rustc` may take advantage of parallelism. `rustc` will coordinate | ||
| with the build system calling it if a [GNU Make jobserver] is passed in the | ||
| `MAKEFLAGS` environment variable. |
There was a problem hiding this comment.
Perhaps it would be good to mention what will happen if a jobserver cannot be found? Maybe something like:
Suggested change
| `MAKEFLAGS` environment variable. | |
| `MAKEFLAGS` environment variable. If a jobserver is not defined, then `rustc` will use a fixed limit on the maximum number of threads. |
There was a problem hiding this comment.
Yeah, not sure: I considered doing that, but does rustc want to commit to anything here? (i.e. it is always going to be the same?).
In addition, if we mention it, should we say what "fixed limit" means? Should we mention/link to the parallel compiler pages, -Zthreads, -Ccodegen-units, etc.?
There was a problem hiding this comment.
I don't think we want to make any commitments, and I don't think mentioning that rustc will "do it's own thing" in the absence of a jobserver is really making a commitment.
It could be more loosely worded to something like "If a jobserver is not defined, then rustc will choose the number of threads to use." Or whatever wording seems sufficiently vague.
I don't think I would bother mentioning -Zthreads for now, since it is highly experimental, and there isn't really any documentation on it (that I am aware of). I also probably wouldn't bother with codegen-units since I don't think we want to get too explicit about how rustc creates threads.
Just a suggestion, though. I'm fine not mentioning it.
There was a problem hiding this comment.
Makes sense. The "sufficiently vague" option sounded like a good compromise to me, so I added that one, please take a look ("threads" -> "jobs", "defined" -> "passed").
Comment on lines
+7
to
+17
| Starting with Rust 1.76.0, `rustc` will warn if a jobserver appears to be | ||
| available but is not accessible, e.g.: | ||
|
|
||
| ```console | ||
| $ echo 'fn main() {}' | MAKEFLAGS=--jobserver-auth=3,4 rustc - | ||
| warning: failed to connect to jobserver from environment variable `MAKEFLAGS="--jobserver-auth=3,4"`: cannot open file descriptor 3 from the jobserver environment variable value: Bad file descriptor (os error 9) | ||
| | | ||
| = note: the build environment is likely misconfigured | ||
| ``` |
There was a problem hiding this comment.
This seemed a little unusual to call out near the beginning of the chapter. Perhaps a different way to flow this would be to move this down below after the paragraph that says "if the + indicator is removed", and show that as an illustration of what happens when + is removed.
There was a problem hiding this comment.
The reason I put it at the top is that the warning applies to all build systems, i.e. the intention was that we would have subsections for other build systems too if needed, but it isn't clear since we only have one, as you mention (and indeed I did have to do a bit of a dance to make the text work without having to repeat the warning block, which is what I originally had, but it felt too heavy).
On the other hand, it may be true that other build systems do not have this issue (i.e. either they don't use a jobserver, or if they do, they enable it "properly", so perhaps nobody else will actually hit that warning, but it is hard to say). Still, it seemed like conceptually it was something outside GNU Make in particular.
Perhaps it could help adding a subsection called "Integrations" or similar with a small text like "These are recommendations for integration of rustc with different build systems." and then make the GNU Make one a subsubsection of that.
I also thought about adding a trivial one about Cargo, saying something like "Cargo already handles it".
There was a problem hiding this comment.
Pushed a new version with what I meant here so that it is simpler to see.
ojeda
force-pushed
the
rustc-jobserver
branch
from
2834c4c
to
f2661c4
Compare
ojeda
added a commit
to Rust-for-Linux/linux
that referenced
this pull request
Feb 29, 2024
`rustc` (like Cargo) may take advantage of the jobserver at any time (e.g. for backend parallelism, or eventually frontend too). In the kernel, we call `rustc` with `-Ccodegen-units=1` (and `-Zthreads` is 1 so far), so we do not expect parallelism. However, in the upcoming Rust 1.76.0, a warning is emitted by `rustc` [1] when it cannot connect to the jobserver it was passed (in many cases, but not all: compiling and `--print sysroot` do, but `--version` does not). And given GNU Make always passes the jobserver in the environment variable (even when a line is deemed non-recursive), `rustc` will end up complaining about it (in particular in Make 4.3 where there is only the simple pipe jobserver style). One solution is to remove the jobserver from `MAKEFLAGS`. However, we can mark the lines with calls to `rustc` (and Cargo) as recursive, which looks simpler. This is being documented as a recommendation in `rustc` [2] and allows us to be ready for the time we may use parallelism inside `rustc` (potentially now, if a user passes `-Zthreads`). Thus do so. Similarly, do the same for `rustdoc` and `cargo` calls. Finally, there is one case that the solution does not cover, which is the `$(shell ...)` call we have. Thus, for that one, set an empty `MAKEFLAGS` environment variable. Link: rust-lang/rust#120515 [1] Acked-by: Masahiro Yamada <[email protected]> Link: rust-lang/rust#121564 [2] Link: https://lore.kernel.org/r/[email protected] [ Reworded to add link to PR documenting the recommendation. ] Signed-off-by: Miguel Ojeda <[email protected]>
petrochenkov
reviewed
Mar 1, 2024
bertschingert
pushed a commit
to bertschingert/bcachefs
that referenced
this pull request
Mar 9, 2024
`rustc` (like Cargo) may take advantage of the jobserver at any time (e.g. for backend parallelism, or eventually frontend too). In the kernel, we call `rustc` with `-Ccodegen-units=1` (and `-Zthreads` is 1 so far), so we do not expect parallelism. However, in the upcoming Rust 1.76.0, a warning is emitted by `rustc` [1] when it cannot connect to the jobserver it was passed (in many cases, but not all: compiling and `--print sysroot` do, but `--version` does not). And given GNU Make always passes the jobserver in the environment variable (even when a line is deemed non-recursive), `rustc` will end up complaining about it (in particular in Make 4.3 where there is only the simple pipe jobserver style). One solution is to remove the jobserver from `MAKEFLAGS`. However, we can mark the lines with calls to `rustc` (and Cargo) as recursive, which looks simpler. This is being documented as a recommendation in `rustc` [2] and allows us to be ready for the time we may use parallelism inside `rustc` (potentially now, if a user passes `-Zthreads`). Thus do so. Similarly, do the same for `rustdoc` and `cargo` calls. Finally, there is one case that the solution does not cover, which is the `$(shell ...)` call we have. Thus, for that one, set an empty `MAKEFLAGS` environment variable. Link: rust-lang/rust#120515 [1] Acked-by: Masahiro Yamada <[email protected]> Link: rust-lang/rust#121564 [2] Link: https://lore.kernel.org/r/[email protected] [ Reworded to add link to PR documenting the recommendation. ] Signed-off-by: Miguel Ojeda <[email protected]>
bertschingert
pushed a commit
to bertschingert/bcachefs
that referenced
this pull request
Mar 9, 2024
`rustc` (like Cargo) may take advantage of the jobserver at any time (e.g. for backend parallelism, or eventually frontend too). In the kernel, we call `rustc` with `-Ccodegen-units=1` (and `-Zthreads` is 1 so far), so we do not expect parallelism. However, in the upcoming Rust 1.76.0, a warning is emitted by `rustc` [1] when it cannot connect to the jobserver it was passed (in many cases, but not all: compiling and `--print sysroot` do, but `--version` does not). And given GNU Make always passes the jobserver in the environment variable (even when a line is deemed non-recursive), `rustc` will end up complaining about it (in particular in Make 4.3 where there is only the simple pipe jobserver style). One solution is to remove the jobserver from `MAKEFLAGS`. However, we can mark the lines with calls to `rustc` (and Cargo) as recursive, which looks simpler. This is being documented as a recommendation in `rustc` [2] and allows us to be ready for the time we may use parallelism inside `rustc` (potentially now, if a user passes `-Zthreads`). Thus do so. Similarly, do the same for `rustdoc` and `cargo` calls. Finally, there is one case that the solution does not cover, which is the `$(shell ...)` call we have. Thus, for that one, set an empty `MAKEFLAGS` environment variable. Link: rust-lang/rust#120515 [1] Acked-by: Masahiro Yamada <[email protected]> Link: rust-lang/rust#121564 [2] Link: https://lore.kernel.org/r/[email protected] [ Reworded to add link to PR documenting the recommendation. ] Signed-off-by: Miguel Ojeda <[email protected]>
bertschingert
pushed a commit
to bertschingert/bcachefs
that referenced
this pull request
Mar 12, 2024
`rustc` (like Cargo) may take advantage of the jobserver at any time (e.g. for backend parallelism, or eventually frontend too). In the kernel, we call `rustc` with `-Ccodegen-units=1` (and `-Zthreads` is 1 so far), so we do not expect parallelism. However, in the upcoming Rust 1.76.0, a warning is emitted by `rustc` [1] when it cannot connect to the jobserver it was passed (in many cases, but not all: compiling and `--print sysroot` do, but `--version` does not). And given GNU Make always passes the jobserver in the environment variable (even when a line is deemed non-recursive), `rustc` will end up complaining about it (in particular in Make 4.3 where there is only the simple pipe jobserver style). One solution is to remove the jobserver from `MAKEFLAGS`. However, we can mark the lines with calls to `rustc` (and Cargo) as recursive, which looks simpler. This is being documented as a recommendation in `rustc` [2] and allows us to be ready for the time we may use parallelism inside `rustc` (potentially now, if a user passes `-Zthreads`). Thus do so. Similarly, do the same for `rustdoc` and `cargo` calls. Finally, there is one case that the solution does not cover, which is the `$(shell ...)` call we have. Thus, for that one, set an empty `MAKEFLAGS` environment variable. Link: rust-lang/rust#120515 [1] Acked-by: Masahiro Yamada <[email protected]> Link: rust-lang/rust#121564 [2] Link: https://lore.kernel.org/r/[email protected] [ Reworded to add link to PR documenting the recommendation. ] Signed-off-by: Miguel Ojeda <[email protected]>
jannau
pushed a commit
to jannau/linux
that referenced
this pull request
Mar 25, 2024
`rustc` (like Cargo) may take advantage of the jobserver at any time (e.g. for backend parallelism, or eventually frontend too). In the kernel, we call `rustc` with `-Ccodegen-units=1` (and `-Zthreads` is 1 so far), so we do not expect parallelism. However, in the upcoming Rust 1.76.0, a warning is emitted by `rustc` [1] when it cannot connect to the jobserver it was passed (in many cases, but not all: compiling and `--print sysroot` do, but `--version` does not). And given GNU Make always passes the jobserver in the environment variable (even when a line is deemed non-recursive), `rustc` will end up complaining about it (in particular in Make 4.3 where there is only the simple pipe jobserver style). One solution is to remove the jobserver from `MAKEFLAGS`. However, we can mark the lines with calls to `rustc` (and Cargo) as recursive, which looks simpler. This is being documented as a recommendation in `rustc` [2] and allows us to be ready for the time we may use parallelism inside `rustc` (potentially now, if a user passes `-Zthreads`). Thus do so. Similarly, do the same for `rustdoc` and `cargo` calls. Finally, there is one case that the solution does not cover, which is the `$(shell ...)` call we have. Thus, for that one, set an empty `MAKEFLAGS` environment variable. Link: rust-lang/rust#120515 [1] Acked-by: Masahiro Yamada <[email protected]> Link: rust-lang/rust#121564 [2] Link: https://lore.kernel.org/r/[email protected] [ Reworded to add link to PR documenting the recommendation. ] Signed-off-by: Miguel Ojeda <[email protected]>
happy-thw
pushed a commit
to happy-thw/rust-for-linux
that referenced
this pull request
Apr 2, 2024
`rustc` (like Cargo) may take advantage of the jobserver at any time (e.g. for backend parallelism, or eventually frontend too). In the kernel, we call `rustc` with `-Ccodegen-units=1` (and `-Zthreads` is 1 so far), so we do not expect parallelism. However, in the upcoming Rust 1.76.0, a warning is emitted by `rustc` [1] when it cannot connect to the jobserver it was passed (in many cases, but not all: compiling and `--print sysroot` do, but `--version` does not). And given GNU Make always passes the jobserver in the environment variable (even when a line is deemed non-recursive), `rustc` will end up complaining about it (in particular in Make 4.3 where there is only the simple pipe jobserver style). One solution is to remove the jobserver from `MAKEFLAGS`. However, we can mark the lines with calls to `rustc` (and Cargo) as recursive, which looks simpler. This is being documented as a recommendation in `rustc` [2] and allows us to be ready for the time we may use parallelism inside `rustc` (potentially now, if a user passes `-Zthreads`). Thus do so. Similarly, do the same for `rustdoc` and `cargo` calls. Finally, there is one case that the solution does not cover, which is the `$(shell ...)` call we have. Thus, for that one, set an empty `MAKEFLAGS` environment variable. Link: rust-lang/rust#120515 [1] Acked-by: Masahiro Yamada <[email protected]> Link: rust-lang/rust#121564 [2] Link: https://lore.kernel.org/r/[email protected] [ Reworded to add link to PR documenting the recommendation. ] Signed-off-by: Miguel Ojeda <[email protected]>
jannau
pushed a commit
to jannau/linux
that referenced
this pull request
Apr 8, 2024
`rustc` (like Cargo) may take advantage of the jobserver at any time (e.g. for backend parallelism, or eventually frontend too). In the kernel, we call `rustc` with `-Ccodegen-units=1` (and `-Zthreads` is 1 so far), so we do not expect parallelism. However, in the upcoming Rust 1.76.0, a warning is emitted by `rustc` [1] when it cannot connect to the jobserver it was passed (in many cases, but not all: compiling and `--print sysroot` do, but `--version` does not). And given GNU Make always passes the jobserver in the environment variable (even when a line is deemed non-recursive), `rustc` will end up complaining about it (in particular in Make 4.3 where there is only the simple pipe jobserver style). One solution is to remove the jobserver from `MAKEFLAGS`. However, we can mark the lines with calls to `rustc` (and Cargo) as recursive, which looks simpler. This is being documented as a recommendation in `rustc` [2] and allows us to be ready for the time we may use parallelism inside `rustc` (potentially now, if a user passes `-Zthreads`). Thus do so. Similarly, do the same for `rustdoc` and `cargo` calls. Finally, there is one case that the solution does not cover, which is the `$(shell ...)` call we have. Thus, for that one, set an empty `MAKEFLAGS` environment variable. Link: rust-lang/rust#120515 [1] Acked-by: Masahiro Yamada <[email protected]> Link: rust-lang/rust#121564 [2] Link: https://lore.kernel.org/r/[email protected] [ Reworded to add link to PR documenting the recommendation. ] Signed-off-by: Miguel Ojeda <[email protected]>
herrnst
pushed a commit
to herrnst/linux-asahi
that referenced
this pull request
Apr 27, 2024
`rustc` (like Cargo) may take advantage of the jobserver at any time (e.g. for backend parallelism, or eventually frontend too). In the kernel, we call `rustc` with `-Ccodegen-units=1` (and `-Zthreads` is 1 so far), so we do not expect parallelism. However, in the upcoming Rust 1.76.0, a warning is emitted by `rustc` [1] when it cannot connect to the jobserver it was passed (in many cases, but not all: compiling and `--print sysroot` do, but `--version` does not). And given GNU Make always passes the jobserver in the environment variable (even when a line is deemed non-recursive), `rustc` will end up complaining about it (in particular in Make 4.3 where there is only the simple pipe jobserver style). One solution is to remove the jobserver from `MAKEFLAGS`. However, we can mark the lines with calls to `rustc` (and Cargo) as recursive, which looks simpler. This is being documented as a recommendation in `rustc` [2] and allows us to be ready for the time we may use parallelism inside `rustc` (potentially now, if a user passes `-Zthreads`). Thus do so. Similarly, do the same for `rustdoc` and `cargo` calls. Finally, there is one case that the solution does not cover, which is the `$(shell ...)` call we have. Thus, for that one, set an empty `MAKEFLAGS` environment variable. Link: rust-lang/rust#120515 [1] Acked-by: Masahiro Yamada <[email protected]> Link: rust-lang/rust#121564 [2] Link: https://lore.kernel.org/r/[email protected] [ Reworded to add link to PR documenting the recommendation. ] Signed-off-by: Miguel Ojeda <[email protected]>
herrnst
pushed a commit
to herrnst/linux-asahi
that referenced
this pull request
Apr 27, 2024
`rustc` (like Cargo) may take advantage of the jobserver at any time (e.g. for backend parallelism, or eventually frontend too). In the kernel, we call `rustc` with `-Ccodegen-units=1` (and `-Zthreads` is 1 so far), so we do not expect parallelism. However, in the upcoming Rust 1.76.0, a warning is emitted by `rustc` [1] when it cannot connect to the jobserver it was passed (in many cases, but not all: compiling and `--print sysroot` do, but `--version` does not). And given GNU Make always passes the jobserver in the environment variable (even when a line is deemed non-recursive), `rustc` will end up complaining about it (in particular in Make 4.3 where there is only the simple pipe jobserver style). One solution is to remove the jobserver from `MAKEFLAGS`. However, we can mark the lines with calls to `rustc` (and Cargo) as recursive, which looks simpler. This is being documented as a recommendation in `rustc` [2] and allows us to be ready for the time we may use parallelism inside `rustc` (potentially now, if a user passes `-Zthreads`). Thus do so. Similarly, do the same for `rustdoc` and `cargo` calls. Finally, there is one case that the solution does not cover, which is the `$(shell ...)` call we have. Thus, for that one, set an empty `MAKEFLAGS` environment variable. Link: rust-lang/rust#120515 [1] Acked-by: Masahiro Yamada <[email protected]> Link: rust-lang/rust#121564 [2] Link: https://lore.kernel.org/r/[email protected] [ Reworded to add link to PR documenting the recommendation. ] Signed-off-by: Miguel Ojeda <[email protected]>
|
@petrochenkov Did you have any additional comments or things you would like to see changed? |
|
@ehuss |
Explicitly document that the jobserver may be used by `rustc` and show the warning to increase the chances that this document is found when searching for solutions online. In particular, add a section about the interaction with build systems, which is intended to contain recommendations on how to integrate `rustc` with different built systems. For GNU Make, recommend using the `+` indicator. In addition, add a note about the issue with GNU Make 4.3 since it is important that users realize they should do this even if they do not expect parallelism from `rustc`. Finally, show how to workaround the issue of `$(shell ...)` calls in recursive Make (which e.g. was needed for the Linux kernel). The GNU Make 4.4 case under `--jobserver-style=pipe` is not added since it got fixed after Rust 1.76.0 already (i.e. `rustc` will not warn if it finds the negative file descriptors). For CMake, recommend using `JOB_SERVER_AWARE` and show a workaround using `$(MAKE)` for earlier versions (when using the Makefile generator). From: rust-lang#120515 Cc: @petrochenkov @belovdv @weihanglo @bjorn3 Signed-off-by: Miguel Ojeda <[email protected]>
|
Thanks! Sorry for the slow reviews on this. @bors r+ rollup |
bors
added
S-waiting-on-bors
|
No worries at all, thanks Eric! |
|
☀️ Test successful - checks-actions |
|
Finished benchmarking commit (c987ad5): comparison URL. Overall result: no relevant changes - no action needed@rustbot label: -perf-regression Instruction countThis benchmark run did not return any relevant results for this metric. Max RSS (memory usage)ResultsThis is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.
CyclesThis benchmark run did not return any relevant results for this metric. Binary sizeThis benchmark run did not return any relevant results for this metric. Bootstrap: 673.832s -> 676.596s (0.41%) |
herrnst
pushed a commit
to herrnst/linux-asahi
that referenced
this pull request
May 2, 2024
`rustc` (like Cargo) may take advantage of the jobserver at any time (e.g. for backend parallelism, or eventually frontend too). In the kernel, we call `rustc` with `-Ccodegen-units=1` (and `-Zthreads` is 1 so far), so we do not expect parallelism. However, in the upcoming Rust 1.76.0, a warning is emitted by `rustc` [1] when it cannot connect to the jobserver it was passed (in many cases, but not all: compiling and `--print sysroot` do, but `--version` does not). And given GNU Make always passes the jobserver in the environment variable (even when a line is deemed non-recursive), `rustc` will end up complaining about it (in particular in Make 4.3 where there is only the simple pipe jobserver style). One solution is to remove the jobserver from `MAKEFLAGS`. However, we can mark the lines with calls to `rustc` (and Cargo) as recursive, which looks simpler. This is being documented as a recommendation in `rustc` [2] and allows us to be ready for the time we may use parallelism inside `rustc` (potentially now, if a user passes `-Zthreads`). Thus do so. Similarly, do the same for `rustdoc` and `cargo` calls. Finally, there is one case that the solution does not cover, which is the `$(shell ...)` call we have. Thus, for that one, set an empty `MAKEFLAGS` environment variable. Link: rust-lang/rust#120515 [1] Acked-by: Masahiro Yamada <[email protected]> Link: rust-lang/rust#121564 [2] Link: https://lore.kernel.org/r/[email protected] [ Reworded to add link to PR documenting the recommendation. ] Signed-off-by: Miguel Ojeda <[email protected]>
asahilina
pushed a commit
to AsahiLinux/linux
that referenced
this pull request
May 10, 2024
`rustc` (like Cargo) may take advantage of the jobserver at any time (e.g. for backend parallelism, or eventually frontend too). In the kernel, we call `rustc` with `-Ccodegen-units=1` (and `-Zthreads` is 1 so far), so we do not expect parallelism. However, in the upcoming Rust 1.76.0, a warning is emitted by `rustc` [1] when it cannot connect to the jobserver it was passed (in many cases, but not all: compiling and `--print sysroot` do, but `--version` does not). And given GNU Make always passes the jobserver in the environment variable (even when a line is deemed non-recursive), `rustc` will end up complaining about it (in particular in Make 4.3 where there is only the simple pipe jobserver style). One solution is to remove the jobserver from `MAKEFLAGS`. However, we can mark the lines with calls to `rustc` (and Cargo) as recursive, which looks simpler. This is being documented as a recommendation in `rustc` [2] and allows us to be ready for the time we may use parallelism inside `rustc` (potentially now, if a user passes `-Zthreads`). Thus do so. Similarly, do the same for `rustdoc` and `cargo` calls. Finally, there is one case that the solution does not cover, which is the `$(shell ...)` call we have. Thus, for that one, set an empty `MAKEFLAGS` environment variable. Link: rust-lang/rust#120515 [1] Acked-by: Masahiro Yamada <[email protected]> Link: rust-lang/rust#121564 [2] Link: https://lore.kernel.org/r/[email protected] [ Reworded to add link to PR documenting the recommendation. ] Signed-off-by: Miguel Ojeda <[email protected]>
jannau
pushed a commit
to AsahiLinux/linux
that referenced
this pull request
May 22, 2024
`rustc` (like Cargo) may take advantage of the jobserver at any time (e.g. for backend parallelism, or eventually frontend too). In the kernel, we call `rustc` with `-Ccodegen-units=1` (and `-Zthreads` is 1 so far), so we do not expect parallelism. However, in the upcoming Rust 1.76.0, a warning is emitted by `rustc` [1] when it cannot connect to the jobserver it was passed (in many cases, but not all: compiling and `--print sysroot` do, but `--version` does not). And given GNU Make always passes the jobserver in the environment variable (even when a line is deemed non-recursive), `rustc` will end up complaining about it (in particular in Make 4.3 where there is only the simple pipe jobserver style). One solution is to remove the jobserver from `MAKEFLAGS`. However, we can mark the lines with calls to `rustc` (and Cargo) as recursive, which looks simpler. This is being documented as a recommendation in `rustc` [2] and allows us to be ready for the time we may use parallelism inside `rustc` (potentially now, if a user passes `-Zthreads`). Thus do so. Similarly, do the same for `rustdoc` and `cargo` calls. Finally, there is one case that the solution does not cover, which is the `$(shell ...)` call we have. Thus, for that one, set an empty `MAKEFLAGS` environment variable. Link: rust-lang/rust#120515 [1] Acked-by: Masahiro Yamada <[email protected]> Link: rust-lang/rust#121564 [2] Link: https://lore.kernel.org/r/[email protected] [ Reworded to add link to PR documenting the recommendation. ] Signed-off-by: Miguel Ojeda <[email protected]>
herrnst
pushed a commit
to herrnst/linux-asahi
that referenced
this pull request
May 27, 2024
`rustc` (like Cargo) may take advantage of the jobserver at any time (e.g. for backend parallelism, or eventually frontend too). In the kernel, we call `rustc` with `-Ccodegen-units=1` (and `-Zthreads` is 1 so far), so we do not expect parallelism. However, in the upcoming Rust 1.76.0, a warning is emitted by `rustc` [1] when it cannot connect to the jobserver it was passed (in many cases, but not all: compiling and `--print sysroot` do, but `--version` does not). And given GNU Make always passes the jobserver in the environment variable (even when a line is deemed non-recursive), `rustc` will end up complaining about it (in particular in Make 4.3 where there is only the simple pipe jobserver style). One solution is to remove the jobserver from `MAKEFLAGS`. However, we can mark the lines with calls to `rustc` (and Cargo) as recursive, which looks simpler. This is being documented as a recommendation in `rustc` [2] and allows us to be ready for the time we may use parallelism inside `rustc` (potentially now, if a user passes `-Zthreads`). Thus do so. Similarly, do the same for `rustdoc` and `cargo` calls. Finally, there is one case that the solution does not cover, which is the `$(shell ...)` call we have. Thus, for that one, set an empty `MAKEFLAGS` environment variable. Link: rust-lang/rust#120515 [1] Acked-by: Masahiro Yamada <[email protected]> Link: rust-lang/rust#121564 [2] Link: https://lore.kernel.org/r/[email protected] [ Reworded to add link to PR documenting the recommendation. ] Signed-off-by: Miguel Ojeda <[email protected]>
herrnst
pushed a commit
to herrnst/linux-asahi
that referenced
this pull request
May 30, 2024
`rustc` (like Cargo) may take advantage of the jobserver at any time (e.g. for backend parallelism, or eventually frontend too). In the kernel, we call `rustc` with `-Ccodegen-units=1` (and `-Zthreads` is 1 so far), so we do not expect parallelism. However, in the upcoming Rust 1.76.0, a warning is emitted by `rustc` [1] when it cannot connect to the jobserver it was passed (in many cases, but not all: compiling and `--print sysroot` do, but `--version` does not). And given GNU Make always passes the jobserver in the environment variable (even when a line is deemed non-recursive), `rustc` will end up complaining about it (in particular in Make 4.3 where there is only the simple pipe jobserver style). One solution is to remove the jobserver from `MAKEFLAGS`. However, we can mark the lines with calls to `rustc` (and Cargo) as recursive, which looks simpler. This is being documented as a recommendation in `rustc` [2] and allows us to be ready for the time we may use parallelism inside `rustc` (potentially now, if a user passes `-Zthreads`). Thus do so. Similarly, do the same for `rustdoc` and `cargo` calls. Finally, there is one case that the solution does not cover, which is the `$(shell ...)` call we have. Thus, for that one, set an empty `MAKEFLAGS` environment variable. Link: rust-lang/rust#120515 [1] Acked-by: Masahiro Yamada <[email protected]> Link: rust-lang/rust#121564 [2] Link: https://lore.kernel.org/r/[email protected] [ Reworded to add link to PR documenting the recommendation. ] Signed-off-by: Miguel Ojeda <[email protected]>
john-cabaj
pushed a commit
to UbuntuAsahi/linux
that referenced
this pull request
May 31, 2024
`rustc` (like Cargo) may take advantage of the jobserver at any time (e.g. for backend parallelism, or eventually frontend too). In the kernel, we call `rustc` with `-Ccodegen-units=1` (and `-Zthreads` is 1 so far), so we do not expect parallelism. However, in the upcoming Rust 1.76.0, a warning is emitted by `rustc` [1] when it cannot connect to the jobserver it was passed (in many cases, but not all: compiling and `--print sysroot` do, but `--version` does not). And given GNU Make always passes the jobserver in the environment variable (even when a line is deemed non-recursive), `rustc` will end up complaining about it (in particular in Make 4.3 where there is only the simple pipe jobserver style). One solution is to remove the jobserver from `MAKEFLAGS`. However, we can mark the lines with calls to `rustc` (and Cargo) as recursive, which looks simpler. This is being documented as a recommendation in `rustc` [2] and allows us to be ready for the time we may use parallelism inside `rustc` (potentially now, if a user passes `-Zthreads`). Thus do so. Similarly, do the same for `rustdoc` and `cargo` calls. Finally, there is one case that the solution does not cover, which is the `$(shell ...)` call we have. Thus, for that one, set an empty `MAKEFLAGS` environment variable. Link: rust-lang/rust#120515 [1] Acked-by: Masahiro Yamada <[email protected]> Link: rust-lang/rust#121564 [2] Link: https://lore.kernel.org/r/[email protected] [ Reworded to add link to PR documenting the recommendation. ] Signed-off-by: Miguel Ojeda <[email protected]> (cherry picked from commit f1e6a71 https:/AsahiLinux/linux) Signed-off-by: John Cabaj <[email protected]>
john-cabaj
pushed a commit
to UbuntuAsahi/linux
that referenced
this pull request
Jun 17, 2024
`rustc` (like Cargo) may take advantage of the jobserver at any time (e.g. for backend parallelism, or eventually frontend too). In the kernel, we call `rustc` with `-Ccodegen-units=1` (and `-Zthreads` is 1 so far), so we do not expect parallelism. However, in the upcoming Rust 1.76.0, a warning is emitted by `rustc` [1] when it cannot connect to the jobserver it was passed (in many cases, but not all: compiling and `--print sysroot` do, but `--version` does not). And given GNU Make always passes the jobserver in the environment variable (even when a line is deemed non-recursive), `rustc` will end up complaining about it (in particular in Make 4.3 where there is only the simple pipe jobserver style). One solution is to remove the jobserver from `MAKEFLAGS`. However, we can mark the lines with calls to `rustc` (and Cargo) as recursive, which looks simpler. This is being documented as a recommendation in `rustc` [2] and allows us to be ready for the time we may use parallelism inside `rustc` (potentially now, if a user passes `-Zthreads`). Thus do so. Similarly, do the same for `rustdoc` and `cargo` calls. Finally, there is one case that the solution does not cover, which is the `$(shell ...)` call we have. Thus, for that one, set an empty `MAKEFLAGS` environment variable. Link: rust-lang/rust#120515 [1] Acked-by: Masahiro Yamada <[email protected]> Link: rust-lang/rust#121564 [2] Link: https://lore.kernel.org/r/[email protected] [ Reworded to add link to PR documenting the recommendation. ] Signed-off-by: Miguel Ojeda <[email protected]> (cherry picked from commit f1e6a71 https:/AsahiLinux/linux) Signed-off-by: John Cabaj <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Explicitly document that the jobserver may be used by
rustc, as well as recommend the+indicator for integration ofrustcinto GNU Make.In particular, show the warning to increase the chances that this document is found when searching for solutions online.
In addition, add a note about the issue with GNU Make 4.3 since it is important that users realize they should do this even if they do not expect parallelism from
rustc.Finally, show how to workaround the issue of
$(shell ...)calls in recursive Make (which e.g. was needed for the Linux kernel).The GNU Make 4.4 case under
--jobserver-style=pipeis not added since it got fixed after Rust 1.76.0 already (i.e.rustcwill not warn if it finds the negative file descriptors).From: #120515
Cc: @petrochenkov @belovdv @weihanglo @bjorn3
v2: To be able to use tab characters for the Make examples, add
<!-- ignore-tidy-{check} -->support totidy.v3: Added "Integration with build systems" section to hold the GNU Make one. Added "by clearing the
MAKEFLAGSvariable". Added "aforementioned" so that it is clear we are talking about the warning above.v4: Added CMake subsection. Added a note that
rustcmay be affected by other flags, e.g.CARGO_MAKEFLAGS.v5: Added that
rustcwill choose the number of jobs if a jobserver is not passed.