- 1. Introduction
- 2. Backstory - Flash
- 3. Backstory - latency
- 4. Backstory - latency
- 5. Backstory - no latency!
- 6. Backstory - LSM
- 7. Backstory - LevelDB
- 8. RocksDB Development in Facebook
- 9. RocksDB architecture
- 10. RocksDB what it's not
- 11. Performance
- 12. Where is it used?
- 13. Code/Demo
- 14. Where is RocksDB going?
- 15. What's the plan for rocksdb-node?
- 16. RocksDB & Node.js - Exponential Disruption!
- 17. THANK YOU!
- RocksDB Backstory
- RocksDB & Node.js
- Code/Demo
- Disruptions!
- Flash - massively disruptive
- NVM-E arrays mainstream
- Data migrating
- SSD 200x improvement
- Problem: Write amplification
- Problem: Endurance problems
- Storage-class memory (SCM)
-
LSM (Log-Structured Merge-tree)
-
Patrick O'Neill Paper 1996
-
BigTable, HBase, Cassandra, Influx
-
Make disk access sequential!
- Level was evolutionary, but has problems..
- Not built for server workloads
- Poor performance once db bigger than RAM
- Compaction process insufficient
- Not optimised for Flash
- RocksDB
- Embedded K/V store with point lookups and range scans
- Optimized for fast storage, e.g. Flash and RAM
- Server Side database with full production support
- Scale linearly with number of CPU cores and with storage IOPs
- Elegant tradeoffs of read, write & space amplification
- i.e. The RUM Conjecture
- Pluggable architecture
- Tunable for different workloads on different hardware
- e.g. pluggable compression modules (snappy, zlib, bzip, etc)
- Level style compaction & 'Universal' style
- Features not in Level
- It's not Level any more!
- Not distributed
- Not fault tolerant
- Not replicated
- Not sharded
- It's an embedded database!
- MyRocks
- CockroachDB
- MongoRocks
- LinkedIn, Yahoo, Pinterest, Airbnb &…
- Try support rocksdb API as closely as possible
- Finish the core RocksDB API
- features which require user defined functions i.e. Comparators, Filters, Merge Operators, and SliceTransform
- Move on to the 'utility' APIs i.e. Backup, Geo, Spatial, JSON etc
- Do not go beyond the rocksdb API! i.e. this can be done in other modules..
- Why write a database in node?
- Imagine a world…
- Keep compute close to the data
- IOT
- No Caching
- Analytics
- No SQL
- ETL & ELT
- "Big" data
- ML
- SCS (microservices are dead!)
- Polygot persistence
- More systems, more data, more work
- long long tail