Passive Disaggregated Persistent Memory (pDPM) at USENIX ATC 2020
[USENIX ATC 2020 Paper] [Slide] [Slide-Short] [Talk]
We explore an alternative approach of building Disaggregated Persistent Memory (DPM) by treating storage nodes as passive parties that do not perform any data processing or data management tasks, a model we call Passive Disaggregated Persistent Memory, or pDPM.
pDPM lowers owning and energy cost, also avoids storage node being the processing scalability bottleneck. pDPM is an instance of passive disaggregation approach and has largely been overlooked in the past. Our work does a thorough exploration of this area.
Based on where to process and manage data, we build three pDPM-based key-value stores: pDPM-Direct, pDPM-Cental, and Clover. All of them provide GET/PUT interfaces and have been tested against YCSB workload.
pDPM is an architecture, which could support many types of usages and applications. The KVS is just one way of utilizing pDPM architecture. It is also possible to build a system on top of pDPM architecture that simply exposes memory interface, it is also possible build a more complex system exposing, e.g., SQL interface.
All systems in this repository are userspace programs, hence no special kernel modifications are necessary. All of them run on top of any popular Linux distributions with the following dependencies:
- libibverbs
- memcached
- numactl
- C++ boost coroutine
For hardware, each machine should have a RDMA NIC card (e.g., Mellanox ConnectX-5) and connected via a Infiniband or Ethernet switch. All systems are able to run on both RoCE and Infiniband mode. If you do not have such testbed, consider using CloudLab.
The testing flow of all systems is almost the same: a) start a server instance, b) start a set of simulated passive memory instances, and c) start a set of compute instances. All systems use memcached as a centralized metadata store to exchange QP number and RKEY information.
Clover is a vanilla development effort. pDPM-Central and pDPM-Direct are built on top of a high-performance two-sided KVS called HERD. For setup tutorials, please refer to the following documents:
The entire code base was designed and implemented by Dr. Shin-Yeh Tsai during his doctoral research at Purdue University.
Shin-Yeh Tsai, Yizhou Shan, Yiying Zhang.
For more disaggregation-related research, make sure to check out our publication list at WukLab.io.
To cite our pDPM paper, please consider using the following bibtex:
@inproceedings {ATC20-pDPM,
title = {{Disaggregating Persistent Memory and Controlling Them Remotely: An Exploration of Passive Disaggregated Key-Value Stores}},
author = {Shin-Yeh Tsai and Yizhou Shan and Yiying Zhang},
booktitle = {2020 {USENIX} Annual Technical Conference ({USENIX} {ATC} 20)},
year = {2020},
url = {https://www.usenix.org/conference/atc20/presentation/tsai},
publisher = {{USENIX} Association},
month = jul,
}