zoyi/bingodb

BingoDB

BingoDB is light-weight and linearly scalable pre-defined (for now) in-memory database. BingoDB provides callback function when TTL expires.

Motivation

DynamoDB is fast but it does not provide callback feature, and other SQL-based services are not fast enough (since accessing disk is slow relative to accessing memory). We specifically wanted to have two features: a database notifies after a event expires in certain period of time, and be able to search with certain index name with high performance. However, we could not find a solid service that satisfied our needs. So we decided to create our own database which provides two key features.

Installation

Install go 1.9 from official website. We only supports 1.9 and higher version of go. Get package manager dep by using command

 go get github.com/golang/dep

and run the following from the project root directory to install dependencies:

dep ensure

Usage

To use BingoDB, you first need to create a {filename}.yml and define of your tables (see example.yml below). Once you've done this, you can simply run a server with following command:

go run cmd/bingodb/bingodb.go -config /path/to/config.yml -addr address:port`

#example.yml
tables:
  #your table name
  onlines:
    fields:
      #define your fields
      #we support string/integer only for now
      id: 'string'
      guestKey: 'string'
      channelId: 'string'
      expiresAt: 'integer'
      lastSeen: 'integer'
    expireKey: 'expiresAt'
    hashKey: 'channelId'
    sortKey: 'id'
    subIndices:
      #name of index you want to search for a particular case
      guest:
        #these fields' value suppose to be in fields
        hashKey: 'channelId'
        sortKey: 'lastSeen'
    metrics:
      #metrics table is optional. If you define metric with ttl and interval with milliseconds,
      #bingoDB records the table state.
      ttl: 600000
      interval: 2000

API Overview

GET /tables

• 존재하는 모든 테이블의 정보를 주는 API

GET /tables/:table?hash=[hash]&sort=[sort]

• 해당 table 에서 hashKey가 hash, sortKey가 sort 인 item을 찾는 API

DELETE /tables/:table?hash=[hash]&sort=[sort]

• 해당 table 에서 hashKey가 hash, sortKey가 sort 인 item을 지우는 API

PUT /tables/:table/

• 새로운 document를 추가하는 API
• $setOnInsert는 해당 document가 디비에 없어 새로 추가되는 경우에만 값을 set하게 됨
• Response에는 디비에 이전 값이 있다면 이전 값과 새로 추가된 값, 교체된 여부를 알 수 있음
• Request example

{
  "$set": {
    "id": "soc1",
    "personKey": "user1",
    "updatedAt": 1505200000000,
    "expiresAt": 1513008702000
  },
  "$setOnInsert": {
    "createdAt": 1505200000000
  }
}

GET /tables/:table/info

• 해당 table 에 대한 정보를 주는 API

GET /tables/:table/scan?hash=[hash]&since=[since]&limit=[limit]&backward=[backward]

• 해쉬 값에 해당하는 아이템들을 list로 얻는 API
• since 값을 포함해 그 이후 데이터를 조회함(backward 값이 1일 경우 그 이전)
• 최대 limit 개수 만큼 조회

GET /tables/:table/indices/:index?hash=[hash]&sort=[sort]

• index 이름을 가진 서브 인덱스에 대해 hashKey가 hash, sortKey가 sort 인 아이템을 찾는 API

GET /tables/:table/indices/:index/scan?hash=[hash]&limit=[limit]&since=[since1]&since=[since2]&since=[since3]&backward=[backward]

• index 이름을 가진 서브 인덱스에 대해 해당하는 아이템들을 list로 얻는 API
• since 값을 포함해 그 이후 데이터를 조회함(backward 값이 1일 경우 그 이전)
• since1: subIndex sort key, since2: primary hash key, since3: primary sort key
• 최대 limit 개수 만큼 조회

Performance

• put: O(lg(n))
• lookup: O(lg(n))
• delete: O(lg(n))
• fetch with since, limit(m): O(lg(n)*m)

Milestones

• Support distributed computing
• Support Fast concurrent lock-free binary search tree
• In-memory database and do not support persistency (completed)
• One bingoDB contains multiple tables (completed)
• A table is structure to store data which contains key/value pair (completed)
• Support JSON format for value type (completed)
• Support dynamic table configuration like NoSQL
• Support pre-define table configuration with config file (completed)
• Provide time to live feature on a value, and the value suppose to be deleted after expired
• Support duplicated TTL value for keys (completed)
• Expired data transits to Message Queue (AMQP, RabbitMQ, etc)
• One table can have multiple secondary indexes (completed)

Benchmarks

• TBD