Gadget: A New Benchmark Harness for Systematic and Robust Evaluation of Streaming State Stores
This repository contains instructions for reproducing the experiments in our EuroSys'22 paper "A New Benchmark Harness for Systematic and Robust Evaluation of Streaming State Stores".
Gadget is a benchmark for an accurate evaluation of KV stores for stateful streaming applications. Gadget supports one or more configurable data sources and simulates the internal operations of a stream processing system to generate realistic state access workloads. Gadget generates and issues state access requests to the KV store while collecting performance measurements on latency and throughput. Gadget currently supports four KV stores with different design and performance characteristics: RocksDB, Lethe, FASTER, and BerkeleyDB.
Table of Contents
- Description & Requirements
- Reproducing the EuroSys22 results
- Install and Run Gadget
- Configure Gadget
- Experiments with Gadget
- Gadget Source Code
- Conference Paper
- License
Project Structure
> tree .
├── configs # Instructions on configuring Gadget
├── experiments # Instructions on performing experiments with Gadget
├── replayer # replay ycsb trace on KV stores
├── reproduceEuroSysResults # reproducing EuroSys22 paper results
├── src # Gadget Source Code
├── trces # Google borg traces
└── tests # Unit tests
Description & Requirements
How to access
Gadget can be downloaded from the following link: https://github.com/CASP-Systems-BU/Gadget
Hardware dependencies
We used the following hardware for our experiments: We run all experiments on a dual-socket machine equipped with 12-core Intel Xeon 4116 CPU running at 2.1 GHz, 32 GB of RAM, and 512GB PC400 NVMe(SK hynix) hard disk.
Software dependencies
We use Ubuntu 20.04 (Linux kernel version5.4). To evaluate a KV store performance using Gadget, you need to install the KV store on your system. To make this step straightforward, we provide a container that already has all KV stores discussed in this paper installed.
Traces
In our experiments, we use traces from Google's Borg system. These traces can be found here
Reproducing the EuroSys22 results
For reviewers, we have prepared a CloudLab profile and a container image (with Gadget and KV stores). Please see a detailed description of reproducing the results presented at the EuroSys22 conference here.
Install and Run Gadget
If you wish to compile Gadget and try it out on your system. You can follow the following steps:
- install build essentials (e.g., compiler):
sudo sh install_build_essentals.sh
- compile Gadget:
git clone git@github.com:CASP-Systems-BU/Gadget.git
cd Gadget
mkdir build
cd build
cmake ..
make
cd src
-
To run gadget, you can use one of our config files provided in
configs
folder(here).Run Gadget by:
./gadget config.txt
.This example generates the state access workload for a tumbling window operator. The results go to a file named
gadget.log
. Please note that to evaluate a KV store using Gadget, the KV store must be installed on the system. (please see here for installing KV stores).
Configure Gadget
Please see here for a detailed description of the Gadget configuration.
Experiments with Gadget
Please see here for a detailed description of performing experiments with Gadget.
Gadget Source Code
Please see here for a detailed description of the Gadget source code.
Conference Paper
[EuroSys] A New Benchmark Harness for Systematic and Robust Evaluation of Streaming State Stores. Esmail Asyabi, Yuanli Wang, John Liagouris, Vasiliki Kalavri, Azer Bestavros. Proceedings of the 17th European Conference on Computer Systems, Rennes, France, April, 2022.
License
Gadget is released under Apache License 2.0