Sarama Heroku

Overview

sarama-heroku is a Go library that makes it easy to connect to Apache Kafka on Heroku. We handle all the certificate management and configuration so that you can start up your Kafka consumers and producers with minimal effort.

Installation

go get -u github.com/deadmanssnitch/sarama-heroku

Heroku

Make sure you have the Heroku CLI plugin installed:

heroku plugins:install heroku-kafka

Next, you'll need to provision a new Kafka add-on or attach an existing one to your app. To provision run:

heroku addons:create heroku-kafka:basic-0 -a [app]
heroku kafka:wait -a [app]

Consumers

Now you are ready to start using the library.

Create a cluster consumer config like the following:

kfkCfg, err := heroku.NewConfig()

config := cluster.NewConfig()
config.ClientID = "app-name." + os.Getenv("DYNO")
config.Net.TLS.Enable = kfkCfg.TLS()
config.Net.TLS.Config = kfkCfg.TLSConfig()

groupID := kfkCfg.Prefix("group-id")
topics := []string{kfkCfg.Prefix("topic")}

consumer, err := cluster.NewClusterConsumer(groupID, topics, config)

❗ Multi-tenant plans require creating the consumer groups before you can use them.

heroku kafka:consumer-groups:create 'group-id' -a [app]

Producers

Furthermore, a producer can be either Sync or Async. Read up on the differences here.

Creating an async producer from a custom config:

kfkCfg, err := heroku.NewConfig()

config := sarama.NewConfig()
config.Producer.Return.Errors = true
config.Producer.RequiredAcks = sarama.WaitForAll
config.Net.TLS.Enable = kfkCfg.TLS()
config.Net.TLS.Config = kfkCfg.TLSConfig()

producer, err := sarama.NewAsyncProducer(kfkCfg.Brokers(), config)

❗ Multi-tenant plans require adding the KAFKA_PREFIX when sending messages. You should use heroku.AppendPrefixTo("topic") to ensure it's set.

producer <- &sarama.ProducerMessage{
  Topic: heroku.AppendPrefixTo("events"),
  Key:   sarama.StringEncoder(key),
  Value: []byte("Message"),
}

For more information about how to set up a config see the sarama documentation.

Multiple Kafka Instances

Use heroku.NewConfigWithName to build a config for a named Kafka instance.

kfkCfg, err := heroku.NewConfigWithName("ONYX")

Environment

Sarama Heroku depends on the following environment variables that are set by the Heroku Kafka add-on:

KAFKA_CLIENT_CERT
KAFKA_CLIENT_CERT_KEY
KAFKA_TRUSTED_CERT
KAFKA_PREFIX (only multi-tenant plans)
KAFKA_URL

Contributing

Thank you so much for your interest in contributing to this repository. We appreciate you and the work you're doing on this SO much.

For details see CONTRIBUTING.md

Thanks

This package was extracted from Dead Man's Snitch, a dead simple monitoring service for Cron jobs and system liveness.