A group of GoLang example templates for writing Apache Kafka production ready applications. The templates were created to be used as a starting point for microservices that utilize Kafka for Event Sourcing as an eventstore. An eventstore is the backbone of an event-sourced mircroservice architecture. Utilizing best practices in GoLang for producers and consumers is therefore critical to a successful project.
Each template does not have any domain specific logic. In some cases wordcount is implemented just to show were to place logic and how to do it in a thread safe fashion. The goal is that new developers can leverage the boiler plate code examples to add additional functionality to an application.
-
consumer-wordcount is a minimal implementation of a Kafka consumer that provides:
- Graceful shutdown to avoid data loss and unexpected behavior
- Configuration via environment variables
- Syslog logging of information and issues
- Implements a word count logic for a basic accumulator of messages
-
consumer-cluster-wordcount is a minimal implementation of a
- Kafka Consumer groups allowing for fault tolerance of consumer clients and load-balancing.
-
producer-wordcount-api is a minimal implementation of a Kafka producer for sending http post vars into a a Kafka topic.
- Graceful shutdown to avoid data loss and unexpected behavior
- Configuration via environment variables
- Syslog logging of information and issues
It assumed that you have an understanding of Apache Kafka and have a version of Kafka running that you can connect to for development. The Devoxx video Introduction to Apache Kafka by James Ward is a very concise way to get up to speed with terminology quickly.
The Kafka quickstart tutorial can assist you in running a local copy. All examples require Kafka v0.9+ to utilize Kafka based client tracking rather than zookeeper.
On macOS with Homebrew installed the following starts a local copy of Kafka
localhost:~ me$ Brew install Kafka
localhost:~ me$ zookeeper-server-start /usr/local/etc/kafka/zookeeper.properties & kafka-server-start /usr/local/etc/kafka/server.properties
The code requires Sarama an MIT-licensed Go client library for Apache Kafka. We utilize this library because it is a pure go implementaion of the wire transfer protocol of Kafka with no C dependencies. It also takes advantage of consumer groups which ensures that for replicated consumers duplicated execution of published messages doesn't occur.
localhost:~ me$ go get github.com/Shopify/sarama
Clustered consumer examples utilize Sarama-Cluster a cluster Sarama extension enabling users to consume topics across balanced nodes. It's important to read the Client-side Assignment Proposal to understand use cases for a clustered consumer.
localhost:~ me$ go get github.com/bsm/sarama-cluster
Please do not fork the repository unless you are submitting a pull request to update the template. For your own usage please mirror the repo.
Service configuration should be done with functional options.
- Functional options for friendly APIs a dotGo presentation by Dave Cheney. Watch the presentation
- Self-referential functions and the design of options by Rob Pike
These templates follow The Twelve Factors a popular and widely used methodology for building web applications.
localhost:~ me$ tail -f /var/log/system.log | grep "serviceName"
This work is built on top of the incredible blog posts from Oscar Oranagwa