DELI: a Log-Structured Secondary Index for HBase/NoSQL
Introduction
DELI
is a secondary index for NoSQL systems. It currently supports global indexing and is applicable for HBase alike NoSQL systems where write performance is optimized through a LSM tree structure [link].
DELI stands for "DEferred Lightweight Indexing". Its unique design (given there are other secondary indices for HBase and NoSQL systems) is that:
- It strictly follows the log-structured design principle that all writes must be append-only.
- It couples the index-base-table sync-up with the compaction.
By this design, DELI
preserves the performance characteristic of original HBase (i.e. write optimized) while adding support for secondary index. Details can be found in the referenced research paper below, which is published in [CCGrid 2015].
Demo
-
Install
docker
on your machine; [link] -
Launch the
docker
container runningDELI
image.sudo docker run -i -t tristartom/deli-hadoop-hbase-ubuntu /bin/bash
-
Inside the container's bash, run the following to demonstrate a DELI client program.
#step 0 source ~/.profile #step 1: first start hdfs cd ~/app/hadoop-2.6.0 bin/hdfs namenode -format sbin/start-dfs.sh #step 2: then run hbase cd ~/app/hbase-0.99.2 bin/start-hbase.sh #step 3: run deli demo program cd ~/app/deli ant #compile the deli client ./tt_sh/run.sh #demonstrate data can be accessed through a value-based Get (GetByIndex).
If you observe Result is key1
by the end of printout, it means the demo runs successfully. The demo source code can be found in ~/app/deli/src/tthbase/client/Demo.java
Reference
"Deferred Lightweight Indexing for Log-Structured Key-Value Stores", Yuzhe Tang, Arun Iyengar, Wei Tan, Liana Fong, Ling Liu, in Proceedings of the 15th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid 2015), Shenzhen, Guangdong, China, May 2015, [pdf]