/unikv

Primary LanguageC++BSD 3-Clause "New" or "Revised" LicenseBSD-3-Clause

UniKV: Toward High-Performance and Scalable KV Storage in Mixed Workloads via Unified Indexing

Introduction

Persistent key-value (KV) stores are mainly designed based on the Log-Structured Merge-tree (LSM-tree), which suffer from large read and write amplifications, especially when KV stores grow in size. Existing design optimizations for LSMtree-based KV stores often make certain trade-offs and fail to simultaneously improve both the read and write performance on large KV stores without sacrificing scan performance. We design UniKV, which unifies the key design ideas of hash indexing and the LSM-tree in a single system. Specifically, UniKV leverages data locality to differentiate the indexing management of KV pairs. It also develops multiple techniques to tackle the issues caused by unifying the indexing techniques, so as to simultaneously improve the performance in reads, writes, and scans. Experiments show that UniKV significantly outperforms several state-of-the-art KV stores (e.g., LevelDB, RocksDB, HyperLevelDB, and PebblesDB) in overall throughput under read-write mixed workloads.

Publications

  • Qiang Zhang, Yongkun Li, Patrick P. C. Lee, Yinlong Xu, Qiu Cui, Liu Tang. UniKV: Toward High-Performance and Scalable KV Storage in Mixed Workloads via Unified Indexing. IEEE ICDE 2020.

Overview

The prototype is written in C++ based on LevelDB

Minimal Requirement

Minimal setup to test the prototype:

  • Ubuntu 14.04 LTS
  • 4GB RAM

Test tool

generate workloads by YCSB-C, the C++ version of YCSB

Installation

On Ubuntu 14.04 LTS or above, install

  • C++ compiler: g++ (version 4.8.4 or above)
  • Snappy: libsnappy-dev

$ sudo apt-get update
$ sudo apt-get install g++ libsnappy-dev

Getting the source code of UniKV
$ git clone https://github.com/ustcadsl/unikv.git

Compile UniKV
$ cd UniKV
$ make clean & make

Install UniKV
$ ./buildLink.sh

Testing the Prototype

Getting the source code of YCSB-C modified by Qiang Zhang @USTC
$ git clone https://github.com/zhangqiangUSTC/YCSB-C.git

Build YCSB-C on Ubuntu
$ sudo apt-get install libtbb-dev
$ cd YCSB-C
$ make clean & make

As the driver for Redis is linked by default, change the runtime library path to include the hiredis library by:
$ export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib

Run test script
$ ./run_KVTest.sh

** This project is released by the ADSLab in USTC.