/kafka-consumer

Consumer for Kafka using brod and elixir (production ready)

Primary LanguageElixirMIT LicenseMIT

Kafka consumer

Scalable consumer for Kafka using brod.

Installation

If available in Hex, the package can be installed as:

  • Add kafka_consumer to your list of dependencies in mix.exs:
def deps do
  [{:kafka_consumer, "~> 2.0"}]
end
  • Ensure kafka_consumer is started before your application:
def application do
  [applications: [:kafka_consumer]]
end

Usage

  • Set config or pass default attributes
config :brod,
  clients: [
    kafka_client: [
      endpoints: [{'localhost', 9092}],
      auto_start_producers: true
    ]
  ]
  • Write your own Consumer. Functions handle_async/1 and handle_sync/2 is overridable
defmodule KafkaConsumer.ExampleConsumer do
  use KafkaConsumer.Consumer

  def handle_async(_message) do
    :ok
  end
end
  • Set event handlers in config.
config :kafka_consumer,
  consumers: [
    [client: :kafka_client,
     group_id: "messaging",
     topics: ["message-events", "system-events"],
     callback: KafkaConsumer.ExampleConsumer,
     callback_args: []]
  ]
  • Define KafkaConsumer.Supervisor as child supervisor of your app.
children = [
  supervisor(KafkaConsumer.Supervisor, []),
]
opts = [strategy: :one_for_one, name: YourApplication.Supervisor]
Supervisor.start_link(children, opts)
  • Start your app.