Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

rustc: document the jobserver #121564

Merged
merged 2 commits into from
May 1, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions src/doc/rustc/src/SUMMARY.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
- [What is rustc?](what-is-rustc.md)
- [Command-line Arguments](command-line-arguments.md)
- [Codegen Options](codegen-options/index.md)
- [Jobserver](jobserver.md)
- [Lints](lints/index.md)
- [Lint Levels](lints/levels.md)
- [Lint Groups](lints/groups.md)
Expand Down
86 changes: 86 additions & 0 deletions src/doc/rustc/src/jobserver.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,86 @@
# Jobserver

Internally, `rustc` may take advantage of parallelism. `rustc` will coordinate
with the build system calling it if a [GNU Make jobserver] is passed in the
`MAKEFLAGS` environment variable. Other flags may have an effect as well, such
as [`CARGO_MAKEFLAGS`]. If a jobserver is not passed, then `rustc` will choose
the number of jobs to use.

Starting with Rust 1.76.0, `rustc` will warn if a jobserver appears to be
available but is not accessible, e.g.:

```console
$ echo 'fn main() {}' | MAKEFLAGS=--jobserver-auth=3,4 rustc -
warning: failed to connect to jobserver from environment variable `MAKEFLAGS="--jobserver-auth=3,4"`: cannot open file descriptor 3 from the jobserver environment variable value: Bad file descriptor (os error 9)
|
= note: the build environment is likely misconfigured
```
Comment on lines +9 to +17
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seemed a little unusual to call out near the beginning of the chapter. Perhaps a different way to flow this would be to move this down below after the paragraph that says "if the + indicator is removed", and show that as an illustration of what happens when + is removed.

Copy link
Contributor Author

@ojeda ojeda Feb 28, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The reason I put it at the top is that the warning applies to all build systems, i.e. the intention was that we would have subsections for other build systems too if needed, but it isn't clear since we only have one, as you mention (and indeed I did have to do a bit of a dance to make the text work without having to repeat the warning block, which is what I originally had, but it felt too heavy).

On the other hand, it may be true that other build systems do not have this issue (i.e. either they don't use a jobserver, or if they do, they enable it "properly", so perhaps nobody else will actually hit that warning, but it is hard to say). Still, it seemed like conceptually it was something outside GNU Make in particular.

Perhaps it could help adding a subsection called "Integrations" or similar with a small text like "These are recommendations for integration of rustc with different build systems." and then make the GNU Make one a subsubsection of that.

I also thought about adding a trivial one about Cargo, saying something like "Cargo already handles it".

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pushed a new version with what I meant here so that it is simpler to see.


## Integration with build systems

The following subsections contain recommendations on how to integrate `rustc`
with build systems so that the jobserver is handled appropriately.

### GNU Make

When calling `rustc` from GNU Make, it is recommended that all `rustc`
invocations are marked as recursive in the `Makefile` (by prefixing the command
line with the `+` indicator), so that GNU Make enables the jobserver for them.
For instance:

<!-- ignore-tidy-tab -->

```make
x:
+@echo 'fn main() {}' | rustc -
petrochenkov marked this conversation as resolved.
Show resolved Hide resolved
```

In particular, GNU Make 4.3 (a widely used version as of 2024) passes a simple
pipe jobserver in `MAKEFLAGS` even when it was not made available for the child
process, which in turn means `rustc` will warn about it. For instance, if the
`+` indicator is removed from the example above and GNU Make is called with e.g.
`make -j2`, then the aforementioned warning will trigger.

For calls to `rustc` inside `$(shell ...)` inside a recursive Make, one can
disable the jobserver manually by clearing the `MAKEFLAGS` variable, e.g.:
petrochenkov marked this conversation as resolved.
Show resolved Hide resolved

```make
S := $(shell MAKEFLAGS= rustc --print sysroot)

x:
@$(MAKE) y

y:
@echo $(S)
```

### CMake

CMake 3.28 supports the `JOB_SERVER_AWARE` option in its [`add_custom_target`]
command, e.g.:

```cmake
cmake_minimum_required(VERSION 3.28)
project(x)
add_custom_target(x
JOB_SERVER_AWARE TRUE
COMMAND echo 'fn main() {}' | rustc -
)
```

For earlier versions, when using CMake with the Makefile generator, one
workaround is to have [`$(MAKE)`] somewhere in the command so that GNU Make
treats it as a recursive Make call, e.g.:

```cmake
cmake_minimum_required(VERSION 3.22)
project(x)
add_custom_target(x
COMMAND DUMMY_VARIABLE=$(MAKE) echo 'fn main() {}' | rustc -
)
```

[GNU Make jobserver]: https://www.gnu.org/software/make/manual/html_node/POSIX-Jobserver.html
[`CARGO_MAKEFLAGS`]: https://doc.rust-lang.org/cargo/reference/environment-variables.html
[`add_custom_target`]: https://cmake.org/cmake/help/latest/command/add_custom_target.html
[`$(MAKE)`]: https://www.gnu.org/software/make/manual/html_node/MAKE-Variable.html
4 changes: 3 additions & 1 deletion src/tools/tidy/src/style.rs
Original file line number Diff line number Diff line change
Expand Up @@ -178,6 +178,7 @@ fn contains_ignore_directive(can_contain: bool, contents: &str, check: &str) ->
if contents.contains(&format!("// ignore-tidy-{check}"))
|| contents.contains(&format!("# ignore-tidy-{check}"))
|| contents.contains(&format!("/* ignore-tidy-{check} */"))
|| contents.contains(&format!("<!-- ignore-tidy-{check} -->"))
{
Directive::Ignore(false)
} else {
Expand Down Expand Up @@ -305,7 +306,8 @@ pub fn check(path: &Path, bad: &mut bool) {

let can_contain = contents.contains("// ignore-tidy-")
|| contents.contains("# ignore-tidy-")
|| contents.contains("/* ignore-tidy-");
|| contents.contains("/* ignore-tidy-")
|| contents.contains("<!-- ignore-tidy-");
// Enable testing ICE's that require specific (untidy)
// file formats easily eg. `issue-1234-ignore-tidy.rs`
if filename.contains("ignore-tidy") {
Expand Down
Loading