SplitFS is a file system for Persistent Memory (PM) which is aimed at reducing the software overhead of applications accessing Persistent Memory. SplitFS presents a novel split of responsibilities between a user-space library file system and an existing kernel PM file system. The user-space library file system handles data operations by intercepting POSIX calls, memory mapping the underlying file, and serving the reads and overwrites using processor loads and stores. Metadata operations are handled by the kernel file system (ext4 DAX).
SplitFS introduces a new primitive termed relink to efficiently support file appends and atomic data operations. SplitFS provides three consistency modes,which different applications can choose from without interfering with each other.
Please cite the following paper if you use SplitFS:
SplitFS : Reducing Software Overhead in File Systems for Persistent Memory. Rohan Kadekodi, Se Kwon Lee, Sanidhya Kashyap, Taesoo Kim, Aasheesh Kolli, Vijay Chidambaram. Proceedings of the The 27th ACM Symposium on Operating Systems Principles (SOSP 19). Paper PDF. Bibtex.
This tutorial walks you through the workflow of compiling splitfs, setting up ext4-DAX, compiling an application and running it with ext4-DAX as well as SplitFS, using a simple microbenchmark. The microbenchmark appends 128MB data to an empty file, in chunks of 4KB each, and does an fsync() at the end.
-
Set up SplitFS
$ cd splitfs; make clean; make; cd .. # Compile SplitFS
$ export LD_LIBRARY_PATH=./splitfs
$ export NVP_TREE_FILE=./splitfs/bin/nvp_nvp.tree
- Set up ext4-DAX
$ sudo mkfs.ext4 -b 4096 /dev/pmem0
$ sudo mount -o dax /dev/pmem0 /mnt/pmem_emul
$ sudo chown -R $USER:$USER /mnt/pmem_emul
- Setup microbenchmark
$ cd micro
$ gcc rw_experiment.c -o rw_expt -O3
$ cd ..
- Run microbenchmark with ext4-DAX
$ sync && echo 3 > /proc/sys/vm/drop_caches # Run this with superuser
$ ./micro/rw_expt write seq 4096
$ rm -rf /mnt/pmem_emul/*
- Run microbenchmark with SplitFS
$ sync && echo 3 > /proc/sys/vm/drop_caches # Run this with superuser
$ LD_PRELOAD=./splitfs/libnvp.so micro/rw_expt write seq 4096
$ rm -rf /mnt/pmem_emul/*
- Results. The resultes show the throughput of doing appends on ext4 DAX and SplitFS. Appends are 5.8x faster on SplitFS.
- ext4-DAX:
0.33M appends/sec - SplitFS:
1.92M appends/sec
- ext4-DAX:
-
Low software overhead. SplitFS tries to obtain performance that is close to the maximum provided by persistent-memory hardware. The overhead due to SplitFS software is significantly lower (by 4-12x) than state-of-the-art file systems such as NOVA or ext4 DAX. As a result, performance on some applications is increased by as much as 2x.
-
Flexible guarantees. SplitFS is the only persistent-memory file system that allows simultaneously running applications to receive different guarantees from the file system. SplitFS offers three modes: POSIX, Sync, and Strict. Application A may in Strict mode, obtaining atomic, synchronous operations from SplitFS, while Application B may simultaneously run in POSIX mode and obtain higher performance. This is possible due to the novel split architecture used in SplitFS.
-
Portability and Stability. SplitFS uses ext4 DAX as its kernel component, so it works with any kernel where ext4 DAX is supported. ext4 DAX is a mature, robust code base that is actively being maintained and developed; as ext4 DAX performance increases over time, SplitFS performance increases as well. This is contrast to research file systems for persistent memory, which do not see development at the same rate as ext4 DAX.
splitfs/contains the source code for SplitFS-POSIXdependencies/contains packages and scripts to resolve dependencieskernel/contains the Linux 4.13.0 kernelmicro/contains the microbenchmarkleveldb/contains LevelDB source codersync/contains the rsync source codescripts/contains scripts to compile and run workloads and kernelsplitfs-so/contains the SplitFS-strict shared libraries for running different workloadssqlite3-trace/contains SQLite3 source codetpcc-sqlite/contains TPCC source codeycsb/contains YCSB source code
The Experiments page has a list of experiments evaluating SplitFS(strict, sync and POSIX) vs ext4 DAX, NOVA-strict, NOVA-relaxed and PMFS. The summary is that SplitFS outperforms the other file systems on the data intensive workloads, while incurring a modest overhead on metadata heavy workloads. Please see the paper for more details.
- Ubuntu 16.04 / 18.04
- At least 32 GB DRAM
- At least 4 cores
- Baremetal machine (Not a VM)
- Intel Processor supporting
clflushoptinstruction (Introduced in Intel processor family -- Broadwell). This can be verified withlscpu | grep clflushopt
- kernel: Installing the linux kernel 4.13.0 involves installing bc, libelf-dev and libncurses5-dev. For ubuntu, please run the script
cd dependencies; ./kernel_deps.sh; cd .. - SplitFS: Compiling SplitFS requires installing Boost. For Ubuntu, please run
cd dependencies; ./splitfs_deps.sh; cd ..
SplitFS is under active development.
- The current implementation of SplitFS handles the following system calls:
open, openat, close, read, pread64, write, pwrite64, fsync, unlink, ftruncate, fallocate, stat, fstat, lstat, dup, dup2, execve and clone. The rest of the calls are passed through to the kernel. - The current implementation of SplitFS works correctly for the following applictions:
LevelDB running YCSB, SQLite running TPCC, tar, git, rsync. This limitation is purely due to the state of the implementation, and we aim to increase the coverage of applications by supporting more system calls in the future.
Copyright for SplitFS is held by the University of Texas at Austin. Please contact us if you would like to obtain a license to use SplitFS in your commercial product.
We thank the National Science Foundation, VMware, Google, and Facebook for partially funding this project. We thank Intel and ETRI IITP/KEIT[2014-3-00035] for providing access to Optane DC Persistent Memory to perform our experiments.
Please contact us at rak@cs.utexas.edu or vijayc@utexas.edu with any questions.