Siberite is a simple LevelDB-backed message queue server
(twitter/kestrel, wavii/darner rewritten in Go).
Siberite is a simple message queue server. Unlike in-memory servers like Redis, Siberite is designed to handle queues much larger than what can fit in RAM. And unlike enterprise-level servers such as RabbitMQ, Siberite stores all messages out of process, using goleveldb for persistent storage.
The result is a durable queue server that uses minimal in-resident memory, regardless of the queue size.
Siberite is based on Robey Pointer's Kestrel, a simple, distributed message queue. Like Kestrel, Siberite follows the "No talking! Shhh!" approach to distributed queues: A single Siberite server maintains a set of queues identified by name. Each queue operates as a strictly-ordered FIFO, while querying from multiple Siberite servers results in a loosely-ordered distributed queue. Siberite also supports Kestrel's two-phase reliable fetch. If a client disconnects before confirming that it has processed a message, the message will be handed off to the next client.
Compared to Kestrel and Darner, Siberite is easier to build, maintain, and distribute. It consumes significantly less memory than Kestrel and offers the ability to consume a queue multiple times using durable cursors.
- Durable cursors for multiple reads
- Siberite clients can consume a single source queue multiple times using the
get <queue>.<cursor_name>syntax. - Normally, with the
get <queue>syntax, the returned message is expired and deleted from the queue. - Using the cursor syntax
get <queue>.<cursor_name>, a durable cursor is initialized. It advances with every read without deleting messages from the source queue. There is no limit on the number of cursors per queue. - If you resume reading from the queue using the standard syntax, Siberite will continue deleting already-served messages from the queue head. Any existing cursor that points to an expired message will restart reading from the current queue head on the next read.
- Durable cursors also support two-phase reliable reads. Failed reads for each cursor are stored in the cursor’s persistent queue and served to other cursor readers.
- Fanout queues
- Siberite allows inserting a message into multiple queues simultaneously using the following syntax:
set <queue>+<another_queue>+<third_queue> ...
Siberite performance benchmarks
Make sure your GOPATH is correct
go get github.com/bogdanovich/siberite
cd $GOPATH/src/github.com/bogdanovich/siberite
go get ./...
go build siberite.go
mkdir ./data
./siberite -listen localhost:22133 -data ./data
2015/09/22 06:29:38 listening on 127.0.0.1:22133
2015/09/22 06:29:38 initializing...
2015/09/22 06:29:38 data directory: ./data
or download darwin-x86_64 or linux-x86_64 builds
Siberite follows the same protocol as Kestrel, which is the memcache TCP text protocol.
telnet localhost 22133
Connected to localhost.
Escape character is '^]'.
set work 0 0 10
1234567890
STORED
set work 0 0 2
12
STORED
get work
VALUE work 0 10
1234567890
END
get work/open
VALUE work 0 2
12
END
get work/close
END
stats
STAT uptime 47
STAT time 1443308758
STAT version siberite-0.4.1
STAT curr_connections 1
STAT total_connections 1
STAT cmd_get 2
STAT cmd_set 2
STAT queue_work_items 0
STAT queue_work_open_transactions 0
END
# other commands:
# get work/peek
# get work/open
# get work/close/open
# get work/abort
# get work.cursor_name
# get work.cursor_name/open
# get work.my_cursor/close/open
# set work+fanout_queue
# flush work
# delete work
# flush_all
# quit
- Waiting a given time limit for a new item to arrive /t= (allowed by protocol but does nothing)