`cargo-pgo`

Cargo subcommand that makes it easier to use PGO and BOLT to optimize Rust binaries.

Installation

$ cargo install cargo-pgo

You will also need the llvm-profdata binary for PGO and llvm-bolt and merge-fdata binaries for BOLT.

You can install the PGO helper binary by adding the llvm-tools-preview component to your toolchain with rustup:

$ rustup component add llvm-tools-preview

For BOLT, it's unfortunately more complicated. See below for BOLT installation guide.

BOLT support is currently experimental.

PGO/BOLT workflow

It is important to understand the workflow of using feedback-directed optimizations. Put simply, it consists of three general steps:

Build binary with instrumentation
- Perform a special build of your executable which will add additional instrumentation code to it.
Gather performance profiles
- Run your instrumented binary on representative workloads. The binary will generate profile files on disk which will be then used to optimize the binary.
- Try to gather as much data as possible. Ideally, run your binary for at least a minute or more.
Build an optimized binary using generated profiles
- The compiler will use the generated profiles to build an optimized version of your binary.
- The binary will be optimized with respect to the profiled workloads. If you execute it on a substantially different workload, the optimizations might not work (or they might even make your binary slower!).

Example

Usage

Before you start to optimize your binaries, you should first check if your environment is set up correctly, at least for PGO (BOLT is more complicated). You can do that using the info command:

$ cargo pgo info

PGO

cargo-pgo provides commands that wrap Cargo commands. It will automatically add --release to all wrapped commands, since it doesn't really make sense to perform PGO on debug builds. If you want to pass any commands to cargo itself, pass them after --.

Generate the profiles

First, you need to generate the PGO profiles. You can currently do it in three ways:
- Build an instrumented binary and then run it manually (recommended).
```
$ cargo pgo build
```
  After the binary is built, you should execute it on some workloads. Note that the binary will be located at <target-dir>/<target-triple>/release/<binary-name>.
- Run an instrumented version of your binary.
```
$ cargo pgo run
```
  This command will instrument the binary and then execute it right away.
- Run tests using an instrumented binary.
```
$ cargo pgo test
```
  In this case you do not have to do anything else, the profiles will be generated after the tests finish executing. Note that unless your test suite is really comprehensive, it might be better to create a binary and run it on some specific workloads.
Build an optimized binary

Once you have generated some profiles, you can execute cargo pgo optimize to build an optimized version of your binary.

BOLT

Using BOLT with cargo-pgo is similar to using PGO, however you have to build BOLT manually and support for it is currently in an experimental stage.

BOLT is not supported directly by rustc, so the instrumentation and optimization commands are not directly applied to binaries built by rustc. Instead, cargo-pgo creates additional binaries that you have to use for gathering profiles and executing the optimized code.

Generate the profiles

First, you need to generate the BOLT profiles. To do that, execute the following command:
```
$ cargo pgo bolt build
```
The instrumented binary will be located at <target-dir>/<target-triple>/release/<binary-name>-bolt-instrumented. Execute it on several workloads to gather as much data as possible.
Build an optimized binary

Once you have generated some profiles, you can execute cargo pgo bolt optimize to build an optimized version of your binary. The optimized binary will be named <binary-name>-bolt-optimized.

BOLT + PGO

Yes, BOLT and PGO can even be combined :) To do that, you should first generate PGO profiles and then use BOLT on already PGO optimized binaries. You can do that using the --with-pgo flag:

# Build PGO instrumented binary
$ cargo pgo build
# Run binary to gather PGO profiles
$ ./target/.../<binary>
# Build BOLT instrumented binary using PGO profiles
$ cargo pgo bolt build --with-pgo
# Run binary to gather BOLT profiles
$ ./target/.../<binary>-bolt-instrumented
# Optimize a PGO-optimized binary with BOLT
$ cargo pgo bolt optimize --with-pgo

BOLT installation

Here's a short guide how to compile LLVM with BOLT. You will need a recent compiler, CMake and ninja.

Download LLVM

$ git clone https://github.com/llvm/llvm-project
$ cd llvm-project

(Optional) Checkout a stable version, at least 14.0.0
```
$ git checkout llvmorg-14.0.5
```
Note that BOLT is being actively fixed, so a trunk version of LLVM might actually work better.

Prepare the build

$ cmake -S llvm -B build -G ninja \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_INSTALL_PREFIX=${PWD}/llvm-install \
  -DLLVM_ENABLE_PROJECTS="clang;lld;compiler-rt;bolt"

Compile LLVM with BOLT
```
$ cd build
$ ninja
$ ninja install 
```
The built files should be located at <llvm-dir>/llvm-install/bin. You should add this directory to $PATH to make BOLT usable with cargo-pgo.

Related work

cargo-pgo I basically independently reimplemented this crate. It uses an almost identical approach, but doesn't support BOLT. It's not maintained anymore, I got a permission from its author to (re)use its name.

License

MIT

osiewicz/cargo-pgo

cargo-pgo