Add support for caching Rust crate compilation
Closed this issue · 7 comments
I outlined a plan for this in a cargo issue.
The big items here are:
- make sccache able to detect rustc as a compiler
- Write a hash function for rustc compiler invocations that uses the things listed in that plan as input. We'll need to refactor the cache lookup code slightly since it currently expects to run a preprocessor and hash that output. We will probably want to make the "generate a hash key" bit generic per-compiler, and then provide a default implementation that does what we currently do for C++ compilers, and a separate implementation for rustc.
I've got a pretty good start on this: https://github.com/luser/sccache/commits/rust-support
I pushed some more changes to the rust-support branch and everything seems to be working for caching Rust compilation!
You can try it out by building sccache from that branch, then you can either write a wrapper script that invokes sccache rustc $* or you can hard-link the sccache binary to a file named rustc, and then run RUSTC=/path/to/rustc cargo build, where /path/to/rustc is your wrapper script or hard-link.
I did some testing this morning, and I was not impressed with the speed of an sccache-wrapped build. I put in some more timing in the server logs and found that the bulk of the time is spent running rustc --emit=dep-info. For some of the crates being built as part of my test app, it took almost as long to run that as to actually run the full compilation. For example, on my Linux machine the rand crate takes ~1.1s to run --emit=dep-info, and ~1.9s to run a full compile.
@eddyb, after looking at the -Z time-passes output, said that he thinks rustc is doing too much work for --emit=dep-info, and we could fix that.
With nightly rustc containing @alexcrichton's fix to exit quickly for --emit=dep-info, sccache is actually a speedup for rust compilation!
luser@eye7:/build/read-process-memory$ cargo clean; time PATH=~/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/bin/:$PATH RUSTC=/tmp/bin/rustc cargo build
Compiling libc v0.2.16
Compiling getopts v0.2.14
Compiling log v0.3.6
Compiling rand v0.3.14
Compiling pulldown-cmark v0.0.3
Compiling tempdir v0.3.5
Compiling skeptic v0.5.0
Compiling read-process-memory v0.1.2-pre (file:///build/read-process-memory)
Finished dev [unoptimized + debuginfo] target(s) in 7.36 secs
real 0m7.447s
user 0m0.060s
sys 0m0.044s
luser@eye7:/build/read-process-memory$ cargo clean; time PATH=~/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/bin/:$PATH RUSTC=/tmp/bin/rustc cargo build
Compiling getopts v0.2.14
Compiling log v0.3.6
Compiling libc v0.2.16
Compiling pulldown-cmark v0.0.3
Compiling rand v0.3.14
Compiling tempdir v0.3.5
Compiling skeptic v0.5.0
Compiling read-process-memory v0.1.2-pre (file:///build/read-process-memory)
Finished dev [unoptimized + debuginfo] target(s) in 1.92 secs
real 0m2.060s
user 0m0.056s
sys 0m0.040s
@luser I tried rust-support on my project, and cargo failed to compile target_build_utils dependency:
$ git clone https://github.com/nagisa/target_build_utils.rs
...
$ RUSTC=$HOME/bin/sccache/rustc cargo build
...
error: Could not compile `target_build_utils`.
$ cargo clean && cargo build
Finished dev [unoptimized + debuginfo] target(s) in 11.47 secs
RUSTC=$HOME/bin/sccache/rustc cargo build give only such hint:
Caused by:
process didn't exit successfully: `/home/evgeniy/bin/sccache/rustc --crate-name target_build_utils src/lib.rs --crate-type lib --emit=dep-info,link -C debuginfo=2 --cfg feature="serde_json" --cfg feature="default" -C metadata=f6a4acc1af3932d7 -C extra-filename=-f6a4acc1af3932d7 --out-dir /tmp/target_build_utils.rs/target/debug/deps -L dependency=/tmp/target_build_utils.rs/target/debug/deps --extern serde_json=/tmp/target_build_utils.rs/target/debug/deps/libserde_json-523da6ef63875c40.rlib --extern phf=/tmp/target_build_utils.rs/target/debug/deps/libphf-e3e74ffb62952a96.rlib` (exit code: 254)
@Dushistov thanks for the info! I'll try that out locally and see if I can reproduce.