This documentation contains steps necessary to reproduce the artifacts for our paper titled Opportunities and Limitations of Modern Hardware Isolation Mechanisms.
The general steps include:
- Prepare hardware
- Set up SPEC
- Run experiments
- Verify results
Tip
To simplify the evaluation process, we provide SSH access to all machines with the benchmark environment already set up. We still include the full manual setup process in this document for reference.
We use the following machines to run our experiments:
x86-64: Framework Laptop 13- Intel Core i7-1165G7
- Hyperthreading disabled, frequency pinned to max without turbo boost
aarch64: Pixel 8- Tensor G3 (1x Cortex-X3, 4x Cortex-A715, 4x Cortex-A510)
- Sleep states disabled, frequency pinned to max on Cortex-X3
morello: Morello Development Board
On both aarch64 and morello, we use the out-of-tree pmu_el0_cycle_counter module to enable userspace access to the pmccntr_el0 register.
With root access, run echo "PMCCNTR=1" > /dev/pmuctl.
Tip
On prepared nodes, this step has already been done.
Unmodified copies of the SPEC benchmarks are available at ~/specCPU2006.tar.gz and ~/specCPU2017.tar.gz.
We patch SPEC CPU2006 and CPU2017 to fix compilation errors with newer toolchains (spec2006.patch, spec2017.patch).
Clone this repository, and run just bootstrap-spec-env.
Then, install SPEC under the following subdirectories:
- CPU2006:
worktrees/spec-env/specCPU2006 - CPU2017:
worktrees/spec-env/specCPU2017
After installation, the directories should both contain the MANIFEST file.
Next, run the following commands to apply the patches:
just setup-spec2006
just setup-spec2017
Our configurations are available under spec-configs which will be used in later steps.
Tip
You can inspect what each target does in justfile.
- Build and install Clang
just clang-x86-64- This can take a long time. For convenience, the prepared nodes already have a warm compilation cache.
- Reproduce Figure 12 - NF overheads on varying batch sizes (x86)
just fig12->results/fig12.csv
- Reproduce Figure 8 - Overhead of Safe and Unsafe IPC
just fig8-x86-64->results/x86-64-ideal.csv
- Reproduce SPEC benchmarks (Fig. 3, 6)
just spec2006-x86-64->spec2006/resultjust spec2017-x86-64->spec2017/result
- Build and install Clang
just clang-aarch64- This can take a long time. For convenience, the prepared nodes already have a warm compilation cache.
- Reproduce Figure 8 - Overhead of Safe and Unsafe IPC
just fig8-aarch64->results/aarch64-ideal.csv
- Reproduce Figure 10 - NF overheads on varying batch sizes (AArch64)
just fig10->results/fig10.csv
- Reproduce SPEC benchmarks (Fig. 4, 7)
just spec2006-aarch64->spec2006/result
We run our benchmarks on both the mainline kernel (mainline) and the Morello-enabled kernel (morello).
Important
By default, the prepared node is booted with the Morello-enabled kernel.
To switch between kernels, run sudo reboot-mainline or sudo reboot-morello.
The machine will shut down and be reachable again in around 5 minutes.
Run on Morello-enabled kernel:
- Reproduce Figure 8 - Overhead of Safe and Unsafe IPC
just fig8-morello->results/morello-ideal.csv,results/morello-exswitch.csv
- Reproduce Figure 11 - NF overheads on varying batch sizes (Morello)
just fig11-morello->results/fig11-morello.csv
- Reproduce SPEC benchmarks (Fig. 5)
just spec2006-morello->spec2006/result-morellojust spec2017-morello->spec2017/result-morello
Run on mainline kernel:
- Reproduce Figure 11 - NF overheads on varying batch sizes (Morello)
just fig11-mainline->results/fig11-mainline.csv
- Reproduce SPEC benchmarks (Fig. 5)
just spec2006-morello-mainline->spec2006/result-mainlinejust spec2017-morello-mainline->spec2017/result-mainline
Cross check the results you obtained with the ones on the paper.