rustdoc disregards job limit

[ilammy]

When running doctests (via cargo doc or cargo test) rustdoc will compile each doctest separately, keeping a queue of compilation jobs. Maximum size of the queue seems to be limited to the number of CPUs in the system, but there seems to be no way to adjust that via command line or other configuration (--jobs N is ignored, for example).

This is particularly nasty on my laptop with 8 CPUs and only 8 GB of RAM that are quickly consumed by eight linkers running in parallel with all other stuff. Lying to rustdoc about CPU count by hooking sysconf() helps as a workaround, but this is hardly an elegant solution.

[Mark-Simulacrum]

This might be having a detrimental impact on our Travis builders as well so I've nominated, presumably the fix is relatively straightforward.

[QuietMisdreavus]

Rustdoc runs the compilation as part of the test, and leaves coordination to the test runner - it doesn't synchronize anything on top of it. You can control the amount of concurrent tests with the --test-threads flag, like this:

$ cargo test --doc -- --test-threads 1

[jyn514]

Rustdoc can't fix this on its end, cargo doesn't pass it the number of jobs:

   Doc-tests log
     Running `rustdoc --crate-type lib --test /home/joshua/src/rust/log/src/lib.rs --crate-name log -L dependency=/home/joshua/.local/lib/cargo/target/debug/deps -L dependency=/home/joshua/.local/lib/cargo/target/debug/deps --cfg const_type_id --cfg atomic_cas --cfg has_atomics --cfg srcbuild --extern cfg_if=/home/joshua/.local/lib/cargo/target/debug/deps/libcfg_if-78e41e584d7442e1.rlib --extern log=/home/joshua/.local/lib/cargo/target/debug/deps/liblog-317dcbf283c962fd.rlib --extern serde_test=/home/joshua/.local/lib/cargo/target/debug/deps/libserde_test-3647946855f36b77.rlib --extern sval=/home/joshua/.local/lib/cargo/target/debug/deps/libsval-0040d76711c48027.rlib -C embed-bitcode=no`

Maybe cargo should pass through --jobs as --test-threads? But that's a cargo issue.

[donpellegrino]

Rustdoc can't fix this on its end, cargo doesn't pass it the number of jobs:

   Doc-tests log
     Running `rustdoc --crate-type lib --test /home/joshua/src/rust/log/src/lib.rs --crate-name log -L dependency=/home/joshua/.local/lib/cargo/target/debug/deps -L dependency=/home/joshua/.local/lib/cargo/target/debug/deps --cfg const_type_id --cfg atomic_cas --cfg has_atomics --cfg srcbuild --extern cfg_if=/home/joshua/.local/lib/cargo/target/debug/deps/libcfg_if-78e41e584d7442e1.rlib --extern log=/home/joshua/.local/lib/cargo/target/debug/deps/liblog-317dcbf283c962fd.rlib --extern serde_test=/home/joshua/.local/lib/cargo/target/debug/deps/libserde_test-3647946855f36b77.rlib --extern sval=/home/joshua/.local/lib/cargo/target/debug/deps/libsval-0040d76711c48027.rlib -C embed-bitcode=no`

Maybe cargo should pass through --jobs as --test-threads? But that's a cargo issue.

Has a cargo issue already been filed for this?

[weihanglo]

See also rust-lang/cargo#10702