Cryptographic Capability Computing (C3) Simulator

Warning

The contents of this repository and linked repositories are solely for research purposes and may contain software with vulnerabilities, such as outdated libraries. Do not use in production.

This material is based upon work supported by the Naval Information Warfare Center Pacific and the Defense Advanced Research Project Agency under Prototype Other Transaction Agreement No. N66001-22-9-4017. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the Space and Naval Warfare Systems Center Pacific or the Defense Advanced Research Project Agency.

Table of Contents

Quick Start
Installing Simics and dependencies
- Clone source code
- Download and install required packages
Building and configuring C3 simulator and software
Running a workload in Simics
Regression Testing with PyTest:
Misc
- Pre-commit hooks

Quick Start

This quick start is tested on Ubuntu 20.04. It assumes dependencies, including the Simics simulator, are installed, Simics is installed. To install Simics, see Install Simics. To install the remaining dependencies, see Install other dependencies.

Setting up the C3 simulator

To setup user-space C3 environment:

# Clone repository
mkdir -p [CC_INSTALL_DIRECTORY]
cd [CC_INSTALL_DIRECTORY]
git clone --recurse-submodules https://github.com/IntelLabs/c3-simulator.git .

# Configure Simics project with default SIMICS_BIN=/opt/simics/simics-6/simics-latest/bin
make -f config-user.mk simics_setup
# or, if needed, set SIMICS_BIN explicitly
make -f config-user.mk simics_setup SIMICS_BIN=/some/other/path/to/simics

# Generate checkpoint (built upon /opt/simics/checkpoints/glibc_latest.ckpt, if available)
make ckpt-cc_llvm

# Build C3 Simics modules
make -B

Running simple workloads

To run simple workloads (e.g., microbenchmarks/hello.cpp):

./simics scripts/runworkload_common.simics \
    src_path=microbenchmarks \
    src_file=hello.cpp \
    workload_name=a.out \
    model=c3 \
    checkpoint=checkpoints/cc_llvm.ckpt \
    gcc_flags="-g -gdwarf -Werror -ldl -lm -lpthread -pthread -Iinclude"

Running pre-compiled binaries

clang++ microbenchmarks/hello.cpp -o a.out
./simics scripts/runworkload_common.simics \
    src_path=. \
    src_file=a.out \
    workload_name=a.out \
    model=c3 \
    checkpoint=checkpoints/cc_llvm.ckpt \
    nobuild=TRUE \
    env_vars="LD_LIBRARY_PATH=/home/simics/glibc/glibc-2.30_install/lib"

Running unit tests

To run individual unit tests (.e.g, unit_tests/common/gtest_hello.cpp):

./simics unit_tests/runtest_common.simics \
    src_path=unit_tests/common \
    src_file=gtest_hello.cpp \
    workload_name=a.out \
    model=c3 \
    checkpoint=checkpoints/cc_llvm.ckpt \
    include_folders=unit_tests/include/unit_tests \
    gcc_flags="-g -gdwarf -Werror -ldl -lm -lpthread -pthread -Iinclude -DC3_MODEL=c3"

To run all unit tests:

# Run all unit tests locally on all available models
pytest -v python_tests/test_unit.py --checkpoint checkpoints/cc_llvm.ckpt
# Run all unit tests for specific models (can specify multiple --model args)
pytest -v python_tests/test_unit.py --checkpoint checkpoints/cc_llvm.ckpt --model c3

C3 integrity and intra-object tripwires

The c3_model has initial functional support for integrity checking. By default, only writes are integrity checked, see Using C3 integrity verification for more details.

To enable integrity support, you can pass enable_integrity=1 to the simics scripts. The pytest scripts also support two pseudo modules, c3-integrity and c3-integrity-intra, which will run on the c3, but will enable module and test configuration to exercise C3 integrity checking or C3 integrity checking along with compiler-instrumented intra-object tripwires.

# For single workloads with integrity:
./simics scripts/runworkload_common.simics \
    src_path=microbenchmarks \
    src_file=hello.cpp \
    workload_name=a.out \
    model=c3 \
    enable_integrity=1 \
    checkpoint=checkpoints/cc_llvm.ckpt \
    gcc_flags="-g -gdwarf -Werror -ldl -lm -lpthread -pthread -Iinclude"

# For running pre-compiled binaries with integrity:
clang++ microbenchmarks/hello.cpp -o a.out
./simics scripts/runworkload_common.simics \
    src_path=. \
    src_file=a.out \
    workload_name=a.out \
    model=c3 \
    checkpoint=checkpoints/cc_llvm.ckpt \
    nobuild=TRUE \
    env_vars="LD_LIBRARY_PATH=/home/simics/glibc/glibc-2.30_install/lib:/home/simics/llvm/llvm_install/lib" \
    enable_integrity=1

# For single unit tests with integrity:
./simics unit_tests/runtest_common.simics \
    src_path=unit_tests/common \
    src_file=gtest_hello.cpp \
    workload_name=a.out \
    model=c3 \
    enable_integrity=1 \
    checkpoint=checkpoints/cc_llvm.ckpt \
    include_folders=unit_tests/include/unit_tests \
    gcc_flags="-g -gdwarf -Werror -ldl -lm -lpthread -pthread -Iinclude -DC3_MODEL=c3"

# For unit tests via pytest:
pytest -v python_tests/test_unit.py --checkpoint checkpoints/cc_llvm.ckpt --model c3-integrity

To enable ICV-based intra-object tripwires, you need to use the C3-enabled LLVM/Clang. This can be done by using the custom clang installed on a C3-kernel enabled checkpoint with (see Custom kernel checkpoint):

# For stand-alone workloads with intra-object integrity:
./simics scripts/runworkload_common.simics \
    src_path=microbenchmarks \
    src_file=hello.cpp \
    workload_name=a.out \
    model=c3 \
    enable_integrity=1 \
    checkpoint=checkpoints/cc_kernel.ckpt \
    compiler="/home/simics/llvm/llvm_install/bin/clang++" \
    gcc_flags="-ldl -lm -lpthread -pthread -fuse-ld=lld -finsert-intraobject-tripwires=all"

# For single unit tests with intra-object integrity:
./simics unit_tests/runtest_common.simics \
    src_path=unit_tests/common \
    src_file=gtest_hello.cpp \
    workload_name=a.out \
    model=c3 \
    enable_integrity=1 \
    checkpoint=checkpoints/cc_kernel.ckpt \
    compiler="/home/simics/llvm/llvm_install/bin/clang++" \
    include_folders=unit_tests/include/unit_tests \
    env_vars="LD_LIBRARY_PATH=/home/simics/glibc/glibc-2.30_install/lib:/home/simics/llvm/llvm_install/lib" \
    gcc_flags="-ldl -lm -lpthread -pthread -fuse-ld=lld -finsert-intraobject-tripwires=all -Iinclude -DC3_MODEL=c3"

# For unit tests via pytest:
pytest -v python_tests/test_unit.py --checkpoint checkpoints/cc_kernel.ckpt --have-kernel --model c3-integrity-intra

Note	At present, the `-finsert-intraobject-tripwires` option does not support multiple parallel compilation jobs. If compiling manually (e.g., not with the `runworkload_common.simics` script), make sure to set `-j1` to avoid parallel builds.

Installing Simics and dependencies

The dependencies and installation is for Ubuntu 20.04. For other systems you may have to adapt the instruction here to fit your environment.

Clone source code

git clone --recurse-submodules https://github.com/IntelLabs/c3-simulator.git

Download and install required packages

Install Simics

Download and install Simics v.2023.31 from https://software.intel.com/content/www/us/en/develop/articles/simics-simulator.html.

Create an /opt/simics directory owned by the current user.

The following commands can be executed in a directory where both the Simics package bundle and the package manager archive have been downloaded to install Simics:

tar xf intel-simics-package-manager-1.7.5-linux64.tar.gz
intel-simics-package-manager-1.7.5/ispm packages --install-bundle simics-6-packages-2023-31-linux64.ispm --install-dir /opt/simics/simics-6.0.169 --non-interactive

Next, launch the package manager GUI with intel-simics-package-manager-1.7.5/ispm-gui to associate the needed addons with the Simics base package using the following steps:

When asked for the installation path for packages, enter /opt/simics/simics-6.0.169 and click "Save".
Click the "Addons" tab.
Select "QSP-x86", "QSP-Clear-Linux", and "QSP-CPU".
Click "Save updates".
Close the package manager GUI.

If the addon tab is empty and you encountered a "Unable to load manifest" error during installation, you can navigate to the Platforms tab and manually import the corresponding manifest file from the installation path.

(optional) Install VMP kernel module

The Simics VMP kernel module significantly accelerate simulation. But as is, it may not be compatible with your system kernel or security requirements (e.g., module signing). If possible, it can be installed with:

/opt/simics/simics-6.0.169/bin/vmp-kernel-install

Install other dependencies

On Ubuntu 20.04, initial dependencies can be installed with:

apt install git curl make gcc

The remaining dependencies can be installed with make -f config-user.mk install_dependencies (use make -n to dry-run, as this will use sudo).

Alternatively, you may manually install the following dependencies: bison, curl, flex, git, g++-8, libatk1.0-dev, libatk-bridge2.0-dev, libgtk3-dev, python-3-pip, pytest, and pytest-xdist.

Install updated cmake

To build the LLVM target, the required cmake version is newer than that provided by Ubuntu 20.04. To build locally, use:

make install-cmake

Building and configuring C3 simulator and software

Note	This section, details various build options and configuration for the C3 simulator. Alternatively, you can follow the Quick Start to set up.

Most of the configuration, build, and install commands use Makefiles. You can use make -n <target> to dry-run and view commands make would execute.

Initialize Simics project files and checkpoint

To initialize the Simics project and build additional dependencies, you can run the following commands:

#  To install Simics, download additional dependencies, and extract files
make -f config-user.mk simics_setup

#  NOTE: If needed, set SIMICS_BIN (default: /opt/simics/simics-6/simics-latest/bin)
make -f config-user.mk simics_setup SIMICS_BIN=/some/other/path/bin

#  Create or update CKPT_GLIBC checkpoint (default: checkpoints/cc_glibc.ckpt)
make ckpt-cc_glibc

#  Create or update CKPT_LLVM checkpoint (default: checkpoints/cc_llvm.ckpt)
make ckpt-cc_llvm

#  Build Cryptographic Computing Simics modules
make -B

Alternatively, you can use the old ./setup_and_build.sh, or you can use the -n dry-run flag when running make to inspect commands to run separately.

Custom kernel checkpoint

Linux dependencies are installed along with make install_dependencies. Alternatively, install the following manually: bison, dwarves, flex, libelf-dev, libssl-dev, and llvm.

Create initial Ubuntu checkpoint

To set up an Ubuntu checkpoint with a custom kernel, you first need to create a base Ubuntu checkpoint. Our scripting assumes this checkpoint is based on Ubuntu 20.04, has the following packages installed autoconf cmake ninja-build build-essential patchelf libgtest-dev, and has the simics agent running (see Simics documentation). You can generate such a checkpoint using:

./simics -batch-mode scripts/install_ubuntu.simics

When done, use write-configuration /opt/simics/checkpoints/ubuntu-20.4_latest.ckpt to save a checkpoint. The scripts by default expect to find the checkpoint at /opt/simics/checkpoints/ubuntu-20.4_latest.ckpt, override CKPT_KERNEL_BASE in config-local.mk to use different path.

Note	The script automation relies on external services and may fail in different environments. In this case, you may need to manually install the checkpoint by manually following the steps in `scripts/install_ubuntu.simics`.

For troubleshooting, it is recommended to run with graphical console enabled; the initial boot will be in the VGA view, after which GRUB will configure the serial console and continue installation via that.

Update kernel

This assumes you have an initial kernel checkpoint, if not, see Create initial Ubuntu checkpoint. Once you have an initial ubuntu checkpoint (default: CKPT_KERNEL_BASE=/opt/simics/checkpoints/ubuntu-20.4_latest.ckpt), you can generate a checkpoint with a custom kernel using:

# Set CKPT_KERNEL_BASE in config-local.mk if needed, (default: checkpoints/cc_kernel.ckpt)
make ckpt-cc_kernel

This will create a new checkpoint at checkpoints/cc_kernel.ckpt.GIT_SHA and create/update a symlink to it from checkpoints/cc_kernel.ckpt.

You can also manually update the kernel of an existing checkpoint with the command:

./simics scripts/update_ubuntu_kernel.simics \
        checkpoint=/path/to/ubuntu_checkpoint_to_build_on.ckpt \
        upload_llvm=TRUE \
        upload_glibc=TRUE \
        kernel=linux/linux.tar.gz \
        save_checkpoint=checkpoints/new_checkpoint_name.ckpt

This assumes linux.tar.gz contains a pre-made linux source within the src directory of the package. Note, that this is not necessary if using the C3 packaged configuration and kernel, in which case you can just run make ckpt—cc_kernel as instructed elsewhere.

Cleaning build artifacts and files

# To clean only Simics modules:
make clean
# To clean most build artifacts (e.g., for glibc, llvm, and linux), run:
make mrproper

Neither of the commands will remove checkpoints. To do so, delete the checkpoints folder(s) manually. Note that checkpoints by default are incremental and depend on the originating checkpoint.

Building LLDB with C3-coredump support

To build LLDB with C3-coredump support run:

make make_llvm-lldb

This builds llvm/llvm_install/bin/lldb. It has C3 support for loading variables (and pointers) in C3 cryptographic address format, and will automatically read in C3 keys from a coredump file generated by a C3-enabled process. This requires that the coredump was generate by the C3-enabled custom kernel (e.g., using a cc_kernel checkpoint).

Building Doxygen documentation

The following commands create doxygen documentation for malloc, crypto and modules under doc/doxygen, you can browse the docs by starting from doc/doxygen/html/index.html. The documentation is auto-generated from inline annotations in comments in the source code files themselves.

make documentation

Using C3 integrity verification

The c3_model has initial functional support for integrity checking based on ICV value bound to a virtual address and corresponding expected cryptographic address. By default, only writes are integrity checked, but exact behavior can be configured via Simics model attributes:

c30_0->integrity_break_on_write_mismatch = (default: FALSE)
c30_0->integrity_fault_on_write_mismatch = (default: TRUE)
c30_0->integrity_break_on_read_mismatch = (default: FALSE)
c30_0->integrity_fault_on_read_mismatch = (default: FALSE)
c30_0->integrity_warn_on_read_mismatch = (default: FALSE)

Kernel support for ICV handling is not implemented, hence the virtual-address based ICVs of one C3-enabled application may pollute the virtual addresses of other C3-enabled applications running within the same simulator instance. At present, the ICVs can be programmatically reset in the Simics shell or script by:

c30_0->integrity_icv_reset = TRUE

Alternatively, there is a helper function:, cc_trigger_icv_map_reset, that can be used to trigger ICV reset from within the target (e.g., a program running on the simulation). NOTE: this functionality is exposed without access-control only for testing purposes.

(optional) Configure makefile targets

The code listing below assume default paths, but these can be configured by creating a config-local.mk file or setting the corresponding environment variables. Some relevant variables and their default values are:

# The path to Simics installation bin directory
SIMICS_BIN=$/opt/simics/simics-6/simics-latest/bin

# Path for a base no-kernel checkpoint that is used as the starting point when
# generating checkpoints without custom kernel. If not set, new checkpoints will
# be created from scratch. Default value is ignored if path is not found.
CKPT_NOKERNEL_BASE=/opt/simics/checkpoints/glibc_latest.ckpt

# Path for checkpoint with glibc, this is generated with `make ckpt-cc_glibc`,
# and will be a symlink to tagged checkpoint folder.
CKPT_GLIBC=checkpoints/cc_glibc.ckpt

Similar to CKPT_GLIBC, but includes llvm.
CKPT_LLVM=checkpoints/cc_llvm.ckpt

Running a workload in Simics

./simics [simics_args]  [run_arg1=val1 run_arg2=val2 ...]

Useful simics_args (optional):

-no-win	run simics with GUI windows hidden (can be displayed on demand)
-batch-mode	run in batch mode (will exit with 0 on success or non-zero on error)

Most run scripts are based on the generic template scripts/runworkload_common.simics It supports the following run-time arguments (see default values in the script):

checkpoint	Specifies the checkpoint.
system	Sets the top level module. For QSP use "board" (default), for TGL: "tgl"
compiler	Overrides the compiler for the workload (unless using custom build command). Default: g++
gcc_flags	Additional compiler flags
model	Selects the model to run the workload with. Default: cc, or c3. (Note: lim_disp configures the LIM model to perform data displacement instead of shifting.)
enable_integrity=1	Enable C3 integrity checking
run_args	Specifies additional workload run arguments
env_vars	Overrides environment arguments for the workload run command
build_cmd	Overrides the default build command
run_cmd	Overrides the default run command
pre_run_fixup	Additional bash commands to execute inside Simics before running the workload
debug	Set to 1 to enable Simics module debug printfs
download_bin_path	If defined, the workload binary and the compiled libc will be downloaded to the specified host directory.
disable_meta_check	LIM-only setting. If set to 1, tags and bounds will not be evaluated
break_on_exception	LIM-only setting. If set to 1, will stop simulation on exceptions (excl. Page Fault)
magic	Set to 1 to enable magic breakpoint
mem_profiler	Set to 1 to enable memory profiler
run_cycles=N	If set, the workload will run for N billion cycles and pause. Default: and stop after completion
cache	Set to 1 to enable caching model
exit	Set to 1 to exit on completion (code 0) or error (non-zero code)

Additional run-time arguments for specific scripts: spec/scripts/generic.simics:

spec	Specifies the SPEC workload name.
spec_size	Specifies the SPEC experiment size (test/ref)

Useful examples:

./simics scripts/runworkload_common.simics \
	src_path=microbenchmarks \
	src_file=hello.cpp \
	workload_name=a.out \
	model=cc \
	checkpoint=checkpoints/cc_llvm.ckpt \
	gcc_flags="-g -gdwarf -Werror -ldl -lm -lpthread -pthread -Iinclude"

Regression Testing with PyTest:

The tests are currently configured to use LLVM’s libunwind, consequently you must use an LLVM checkpoint to run unit tests (e.g., checkpoints/cc_llvm.ckpt as described above).

Run all tests (12 jobs in parallel):

pytest -n12 -v python_tests --checkpoint checkpoints/cc_llvm.ckpt [--model native|cc|lim]

Run only spec tests:

pytest -n12 -v python_tests --checkpoint checkpoints/cc_llvm.ckpt[--model native|cc|lim]

# all spec workloads:
pytest -n12 -v python_tests/test_spec.py --checkpoint checkpoints/cc_llvm.ckpt

# specific workloads:
pytest -n12 -v python_tests/test_spec.py --checkpoint checkpoints/cc_llvm.ckpt --spec workload_name [--spec workload_name ...]

Run only unit tests:

pytest -n12 -v python_tests/test_unit.py --checkpoint checkpoints/cc_llvm.ckpt

Common options:

--checkpoint PATH	Set the checkpoint to use
--model	Run tests only with the specified model. Can specify multiple models by appending '--model <model_name>' for each model. The 'c3-integrity' model will run on the c3 but configure it to use integrity
-d	Load-balance tests. Shortcut for '--dist=load'
--have-kernel	Run C3-kernel dependent tests

You can also run individual unit tests:

./simics unit_tests/runtest_common.simics \
	src_path=unit_tests/common \
	src_file=gtest_hello.cpp \
	workload_name=a.out \
	model=cc \
	checkpoint=checkpoints/cc_llvm.ckpt \
	include_folders="unit_tests/include/unit_tests" \
	gcc_flags="-g -gdwarf -Werror -ldl -lm -lpthread -pthread -Iinclude -Iunit_tests/include -DC3_MODEL=cc"

Misc

Pre-commit hooks

To enforce coding guidelines locally, you can install pre-commit hooks that run tests on the staged changes before allowing a commit to pass. To enable default commit hooks, you can run

#  To install, run:
make pre-commit-install
#  To uninstall, run:
make pre-commit-uninstall

The pre-commit hook will apply whitespace fixes automatically to your working tree, you can inspect those changes using git diff, and then add them to your commit. The pre-commit hook also runs clang-format and cpplint checks. You may need to manually address issues reported by cpplint. Cosmetic code style changes can be automatically applied by running clang-format -i <filename>, or without the -i flag to only inspect changes without applying them.

In some cases you may not be able to fix all changes, or you may need to commit files that intentionally violate code style rules. To do so, you can always run git commit --no-verify. However, when possible, avoid disregarding issues.

adwait/c3-simulator