Go is a compiled language with a number of high-performance data stores that can be compiled with Go into a single binary. These databases are called "in-process" or "embedded".
By default, most apps slap MongoDB or MySQL/MariaDB into an application for storing state. Since we left the .net/node/php/ruby/python
world we can have our database bundled right into our application. For smaller projects, which do not need a globally distributed / sharded database, this is a great devops win.
Embedded databases are propbably not for you if:
- building a serverless app (AWS lambda / Zeit Now)
- dealing with >1 Million HTTP requests a day
- have highly concurrent request patterns
These benchmarks and tests are off for a variety of reasons including differing levels of durability/sync and feature-scope. I recommend people default to bbolt
for a general store.
boltdb - https://github.com/boltdb/bolt
Solid, well-tested, development is locked. Each transaction has a consistent view of the data as it existed when the transaction started. Slowest of the bunch. Largest file size. Slowest (due to consistent-data views).
- Blast - full text search and indexing server cluster via Raft
- Storm - ORM-ish layer for database access
bbolt - https://github.com/etcd-io/bbolt
Fork of boltdb to add new features.
- BoltHold - simple querying and indexing layer
- Storm - query-builder, indexes, and struct storage
- Buckets - simple access layer
pogreb - https://github.com/akrylysov/pogreb
Optimized store for random lookups. No support for range/prefix scans. Slow in this benchmark. More...
goleveldb - https://github.com/syndtr/goleveldb
Well tested. Smallest storage size.
- ledisdb - redis-like abstraction over it
badgerdb - https://github.com/dgraph-io/badger
Fastest engine for random lookups and inserts.
- Graph engine dgraph built upon it. More...
- Cete document store
keydb - https://github.com/robaho/keydb
Not tested.
tiedot - https://github.com/HouzuoGuo/tiedot/
Tiedot is a document store, it really can't be compared to these other databases correctly. Included only for loose reference. Super-fast write speeds. Huge storage requirements.
- RocksDB
- LMDB
- hyperleveldb
Not tested. Multiple versions exist, most wrap the C code.
database/sql
driver
no database/sql
driver
Comparison between bvinc/go-sqlite-lite & crawshaw/sqlite.
Number of keys: 5000
Minimum key size: 32, maximum key size: 64
Minimum value size: 128, maximum value size: 1024
Concurrency: 3
Running tiedot benchmark...
Put: 0.022 sec, 226300 ops/sec
Get: 0.001 sec, 3552460 ops/sec
Put + Get time: 0.024 sec
File size: 512.00MB
Running pogreb benchmark...
Put: 0.142 sec, 35293 ops/sec
Get: 0.002 sec, 2489791 ops/sec
Put + Get time: 0.144 sec
File size: 4.25MB
Running goleveldb benchmark...
Put: 0.035 sec, 144902 ops/sec
Get: 0.009 sec, 568586 ops/sec
Put + Get time: 0.043 sec
File size: 3.05MB
Running bolt benchmark...
Put: 15.868 sec, 315 ops/sec
Get: 0.004 sec, 1136989 ops/sec
Put + Get time: 15.873 sec
File size: 8.00MB
Running bbolt benchmark...
Put: 14.710 sec, 339 ops/sec
Get: 0.006 sec, 874240 ops/sec
Put + Get time: 14.716 sec
File size: 8.00MB
- https://tech.townsourced.com/post/boltdb-vs-badger/
- https://www.voltdb.com/blog/2015/04/01/foundationdbs-lesson-fast-key-value-store-not-enough/
- https://petewarden.com/2010/10/01/how-i-ended-up-using-s3-as-my-database/
- https://hackernoon.com/what-i-learnt-from-building-3-high-traffic-web-applications-on-an-embedded-key-value-store-68d47249774f
- https://www.cockroachlabs.com/docs/stable/architecture/storage-layer.html
- https://github.com/kval-access-language/kval-language-specification
- https://github.com/blevesearch/bleve (Video)
- http://roaringbitmap.org/
- https://www.pilosa.com/docs/latest/data-model/
- https://github.com/Xeoncross/keyset
If inserting complex data types (structs, maps, slices...) into a key/value store you must encode/decode them first.
protobuf
provides performance, correctness and interoperability. Requires you to generate protobuf files.gob
is the next fastest format with the smallest data size. More for streaming protocols where you don't want to have to have a pre-shared schema. Poor interoperability.json
the default. Reasonable performance. Highest interoperability.xml
is to be avoided.
- badgerdb, goleveldb, boltdb, bboltdb, buntdb, rocksdb, cznic/kv, map
- badgerdb, goleveldb, boltdb benchmark by zchee.
- badgerdb, goleveldb, boltdb, rocksdb benchmark by dgraph. (High-Quality)
- goleveldb, boltdb, pogreb write/put benchmark by mattn.