/nodejs-dublin-rocksdb-may-2017

Table of Contents

Introduction

  • RocksDB Backstory
  • RocksDB & Node.js
  • Code/Demo
  • Disruptions!

Backstory - Flash

  • Flash - massively disruptive
  • NVM-E arrays mainstream
  • Data migrating
  • SSD 200x improvement
  • Problem: Write amplification
  • Problem: Endurance problems
  • Storage-class memory (SCM)

Backstory - latency

img

Backstory - latency

img

Backstory - no latency!

img

Backstory - LSM

  • LSM (Log-Structured Merge-tree)

  • Patrick O'Neill Paper 1996

  • BigTable, HBase, Cassandra, Influx

  • Make disk access sequential!

    img

Backstory - LevelDB

  • Google - 2011

  • Replacement for SQLite in IndexDB

    img

RocksDB Development in Facebook

  • Level was evolutionary, but has problems..
    • Not built for server workloads
    • Poor performance once db bigger than RAM
    • Compaction process insufficient
    • Not optimised for Flash
  • RocksDB
    • Embedded K/V store with point lookups and range scans
    • Optimized for fast storage, e.g. Flash and RAM
    • Server Side database with full production support
    • Scale linearly with number of CPU cores and with storage IOPs
  • Elegant tradeoffs of read, write & space amplification

RocksDB architecture

  • Pluggable architecture
    • Tunable for different workloads on different hardware
    • e.g. pluggable compression modules (snappy, zlib, bzip, etc)
  • Level style compaction & 'Universal' style
  • Features not in Level
  • It's not Level any more!

img

RocksDB what it's not

  • Not distributed
  • Not fault tolerant
  • Not replicated
  • Not sharded
  • It's an embedded database!

Performance

Where is it used?

  • MyRocks
  • CockroachDB
  • MongoRocks
  • LinkedIn, Yahoo, Pinterest, Airbnb &…

Code/Demo

Where is RocksDB going?

What's the plan for rocksdb-node?

  • Try support rocksdb API as closely as possible
  • Finish the core RocksDB API
    • features which require user defined functions i.e. Comparators, Filters, Merge Operators, and SliceTransform
  • Move on to the 'utility' APIs i.e. Backup, Geo, Spatial, JSON etc
  • Do not go beyond the rocksdb API! i.e. this can be done in other modules..

RocksDB & Node.js - Exponential Disruption!

  • Why write a database in node?
    • Imagine a world…
  • Keep compute close to the data
  • IOT
  • No Caching
  • Analytics
    • No SQL
    • ETL & ELT
    • "Big" data
  • ML
  • SCS (microservices are dead!)
  • Polygot persistence
  • More systems, more data, more work
    • long long tail

THANK YOU!