Temper

About

Temper is a framework for modelling concurrency and failure in distributed systems. The name comes from Temporal Fuzzing, a term coined by Rachel Kroll.

Similar to Loom, programs targeting Temper should be able to switch between simulation mode, and calling real APIs in release mode.

It is in early development, and is not yet more than an experiment. It currently features:

Low level x86/ARM memory models
Rust/C++ 11 memory model

Planned features:

MESI protocol simulation to measure cache line contention and false sharing
Data race detection
TCP/IP, including congestion, asymmetric net splits, and Byzantine faults
Disk operations, including fsync and power failure corruption
SQL transactional isolation

Related Work:

Madsim, a similar project with an emphasis on networking
FoundationDB's testing strategy
TigerBeetle's fault injection testing
Loom, which exhaustively tests on a single node
Timecraft, a distributed system testing tool for WebAssembly

Components

Memlog

Memlog simulates the Rust memory model (C++ 11 without Consume). Combined with operation reordering in Temper, its goal is full coverage. It contains a series of test cases dervied from Preshing on Programming, C++ Concurrency in Action, the C++ Standard, blog posts and many Stack Overflow questions.

Todo:

Detect data races in non-atomic datatypes
Expose API to declare what can be reordered
MESI protocol simulation
Locks
Seeded randomness
Reentry support for fetch_update
Support multiple datatypes

Low Level

Temper contains a low level simulation of x86/ARM memory models. It is intended for experimentation, as the operations cannot be translated to standard Rust calls in release mode.