benthos: A Go repository from javarange

Benthos is a messaging bridge service that supports a wide and growing list of input and output protocols.

A range of internal buffer strategies are available, allowing you to select a balance between latency, protection against back pressure and file based persistence, or nothing at all (direct bridge).

Design

Benthos has inputs, an optional buffer, and outputs, which are all set in a single config file.

+--------------------------------------+
|            Input Stream              |
| ( ZMQ, NSQ, AMQP, Kafka, HTTP, ... ) |--+
+--------------------------------------+  |
                                          v
             +--------------------------------------------+
             |                   Buffer                   |
             | ( Memory-Mapped Files, Memory, None, etc ) |
             +--------------------------------------------+
                             |
                             |  +--------------------------------------+
                             +->|             Output Stream            |
                                | ( ZMQ, NSQ, AMQP, Kafka, HTTP, ... ) |
                                +--------------------------------------+

Supported Protocols

Currently supported input/output targets:

ZMQ4 (PUSH, PULL, SUB, PUB)
Nanomsg/Scalability Protocols (PUSH, PULL, SUB, PUB)
RabbitMQ (AMQP)
NSQ
NATS
Kafka
HTTP 1.1 POST/GET
STDIN/STDOUT
File

You can also have multiple outputs or inputs by choosing a routing strategy (fan in, fan out, round robin, etc.)

For a full and up to date list you can print them from the binary:

# Print inputs, buffers and output options
benthos --print-inputs --print-buffers --print-outputs | less

Install

Build with Go:

go get github.com/jeffail/benthos/cmd/...

Or, download from here.

Run

benthos -c ./config.yaml

Config

Check out the samples in ./config, or create a fully populated default configuration file:

benthos --print-yaml > config.yaml
benthos --print-json > config.json

The configuration file should contain a section for an input, output, and a buffer. For example, if we wanted to output to a ZMQ4 push socket our output section in a YAML config might look like this:

output:
  type: zmq4
  zmq4:
    addresses:
      - tcp://*:1234
    socket_type: PUSH

There are also configuration sections for logging and metrics, if you print an example config you will see the available options.

Speed and Benchmarks

Benthos isn't doing much, so it's reasonable to expect low latencies and high throughput. Here's a table of benchmarks from an 4-core (2.4ghz) machine, bridging messages of 5000 bytes through a 500MB memory buffer via various protocols:

	Avg. Latency (ms)	99th P. Latency (ms)	Msg/s	Mb/s
ZMQ	5.940	67.268	31357	157.383
Nano	3.312	20.020	22894	114.907
HTTP	0.549	8.751	4280	21.479

Take these results with a pinch of salt. I've added some benchmarking utilities in ./cmd/test, hopefully a third party can cook us up some more meaningful figures from a better set up than I have.

ZMQ4 Support

Benthos supports ZMQ4 for both data input and output. To add this you need to install libzmq4 and use the compile time flag when building benthos:

go install -tags "ZMQ4" ./cmd/...

Vendoring

Versions of go above 1.6 should automatically go get all vendored libraries. Otherwise, while cloning use --recursive:

git clone https://github.com/jeffail/benthos --recursive

Or, if the repo is already cloned, get the latest libraries with:

git submodule update --init

To add new libraries simply run:

PACKAGE=github.com/jeffail/util
git submodule add https://$PACKAGE vendor/$PACKAGE"

It might be handy to set yourself a function for this in your .bashrc:

function go-add-vendor {
	git submodule add https://$1 vendor/$1
}

Docker

There's a Dockerfile for creating a benthos docker image. This is built from scratch and so you'll need to build without CGO (CGO_ENABLED=0) for your benthos build to run within it. Create it like this:

CGO_ENABLED=0 make docker
docker run --rm benthos

Then use the image:

docker run --rm -v ~/benthos.yaml:/config.yaml -v /tmp/data:/data -p 8080:8080 \
	benthos -c /config.yaml

javarange/benthos