librdkafka - the Apache Kafka C/C++ client library

https://github.com/edenhill/librdkafka

librdkafka is a C library implementation of the Apache Kafka protocol, containing both Producer and Consumer support. It was designed with message delivery reliability and high performance in mind, current figures exceed 1 million msgs/second for the producer and 3 million msgs/second for the consumer.

librdkafka is licensed under the 2-clause BSD license.

For an introduction to the performance and usage of librdkafka, see INTRODUCTION.md

See the wiki for a FAQ.

NOTE: The master branch is actively developed, use latest release for production use.

Overview

High-level producer
High-level balanced KafkaConsumer (requires broker >= 0.9)
Simple (legacy) consumer
Compression: snappy, gzip, lz4
SSL support
SASL (GSSAPI/Kerberos/SSPI, PLAIN, SCRAM) support
Broker version support: >=0.8 (see Broker version compatibility)
Stable C & C++ APIs (ABI safety guaranteed for C)
Statistics metrics
Debian package: librdkafka1 and librdkafka-dev in Debian and Ubuntu
RPM package: librdkafka and librdkafka-devel
Gentoo package: dev-libs/librdkafka
Portable: runs on Linux, OSX, Win32, Solaris, FreeBSD, ...

Language bindings

C#/.NET: confluent-kafka-dotnet (based on rdkafka-dotnet)
C++: cppkafka
D (C-like): librdkafka
D (C++-like): librdkafkad
Erlang: erlkaf
Go: confluent-kafka-go
Haskell (kafka, conduit, avro, schema registry): hw-kafka
Haskell: haskakafka
Haskell: haskell-kafka
Lua: luardkafka
Node.js: node-rdkafka
Node.js: node-kafka
Node.js: kafka-native
OCaml: ocaml-kafka
PHP: phpkafka
PHP: php-rdkafka
Python: confluent-kafka-python
Python: PyKafka
Ruby: Hermann
Ruby: rdkafka-ruby
Rust: rust-rdkafka
Tcl: KafkaTcl
Swift: Perfect-Kafka

Users of librdkafka

kafkacat - Apache Kafka swiss army knife
Wikimedia's varnishkafka - Varnish cache web log producer
Wikimedia's kafkatee - Kafka multi consumer with filtering and fanout
rsyslog
syslog-ng
collectd
logkafka - Collect logs and send to Kafka
redBorder
Headweb
Produban's log2kafka - Web log producer
fuse_kafka - FUSE file system layer
node-kafkacat
OVH - AntiDDOS
otto.de's trackdrd - Varnish log reader
Microwish has a range of Kafka utilites for log aggregation, HDFS integration, etc.
aidp - kafka consumer embedded Lua scripting language in data process framework
Yandex ClickHouse
NXLog - Enterprise logging system, Kafka input/output plugin.
large unnamed financial institutions
and many more..
Let me know if you are using librdkafka

Usage

Requirements

The GNU toolchain
GNU make
pthreads
zlib (optional, for gzip compression support)
libssl-dev (optional, for SSL and SASL SCRAM support)
libsasl2-dev (optional, for SASL GSSAPI support)

Instructions

Building

  ./configure
  make
  sudo make install

NOTE: See README.win32 for instructions how to build on Windows with Microsoft Visual Studio.

NOTE: See CMake instructions for experimental CMake build (unsupported).

Usage in code

See examples/rdkafka_example.c for an example producer and consumer.

Link your program with -lrdkafka -lz -lpthread -lrt.

Documentation

The public APIs are documented in their respective header files:

The C API is documented in src/rdkafka.h
The C++ API is documented in src-cpp/rdkafkacpp.h

To generate Doxygen documents for the API, type:

make docs

Configuration properties are documented in CONFIGURATION.md

For a librdkafka introduction, see INTRODUCTION.md

Examples

See the examples/sub-directory.

Tests

See the tests/sub-directory.

Support

File bug reports, feature requests and questions using GitHub Issues

Questions and discussions are also welcome on irc.freenode.org, #apache-kafka, nickname Snaps.

Commercial support

Commercial support is available from Edenhill services

adyach/librdkafka