Parallelize some phases of compilation between a crate and its dependency

[steveklabnik]

Issue by brson
Saturday Sep 08, 2012 at 22:00 GMT

For earlier discussion, see rust-lang/rust#3431

This issue was labelled with: A-an-interesting-project, A-build, E-hard, I-wishlist in the Rust repository

---

With a single driver building multiple crates we can start running more parts of the compilation process in parallel.

The basic idea is to output the metadata about a crate before code generation so that the next compile can proceed based on that metadata.

rustc gets modified in a few ways:

• Just before trans we serialize the bulk of the metadata and insert a hook to give the multi-crate driver an opportunity to copy it to another task to begin the next compilation phase. There will still be some metadata that needs to be serialized after trans.
• Insert another hook into crate resulotion for reading the metadata binary. The multi-crate driver will use this to receive the metadata output from previous crates and feed them to the next crates.

The end result is that parsing and analysis will be largely done in parallel with code generation for multi-crate builds.

Beyond that we can also parse all crates in parallel.

This would require that we make sure we are using LLVM in a threadsafe way, unless we can get all the trans passes onto a single thread.

Related to #2237

[hanna-kruppe]

Mostly a duplicate of rust-lang/rust#44587 but that one's more plausible (but still a bad idea IMO) in that it wants to make Cargo manage the metadata/codegen split. As noted over there, "codegen only from metadata" also overlaps with MIR-only rlibs (rust-lang/rust#38913). Parts of this issue are also already obsolete, we have multithreaded LLVM now, though not multi-threaded MIR->LLVM IR codegen yet, and there's active work on parallelizing the latter and the rest of rustc internally (https://internals.rust-lang.org/t/parallelizing-rustc-using-rayon/6606 and more speculatively rust-lang/rust#47518). There's also ideas to remove the crate-at-a-time compilation model in the long term. So besides being a duplicate, this issue has been mostly outpaced by development already and will likely become completely obsolete before anything further happens with it.