librdkafka - the Apache Kafka C/C++ client library

https://github.com/edenhill/librdkafka

librdkafka is a C library implementation of the Apache Kafka protocol, containing both Producer and Consumer support. It was designed with message delivery reliability and high performance in mind, current figures exceed 1 million msgs/second for the producer and 3 million msgs/second for the consumer.

librdkafka is licensed under the 2-clause BSD license.

For an introduction to the performance and usage of librdkafka, see INTRODUCTION.md

See the wiki for a FAQ.

NOTE: The master branch is actively developed, use latest release for production use.

Overview

High-level producer
High-level balanced KafkaConsumer (requires broker >= 0.9)
Simple (legacy) consumer
Compression: snappy, gzip, lz4, zstd
SSL support
SASL (GSSAPI/Kerberos/SSPI, PLAIN, SCRAM) support
Broker version support: >=0.8 (see Broker version compatibility)
Stable C & C++ APIs (ABI safety guaranteed for C)
Statistics metrics
Debian package: librdkafka1 and librdkafka-dev in Debian and Ubuntu
RPM package: librdkafka and librdkafka-devel
Gentoo package: dev-libs/librdkafka
Portable: runs on Linux, OSX, Win32, Solaris, FreeBSD, AIX, ...

Language bindings

C#/.NET: confluent-kafka-dotnet (based on rdkafka-dotnet)
C++: cppkafka
D (C-like): librdkafka
D (C++-like): librdkafkad
Erlang: erlkaf
Go: confluent-kafka-go
Haskell (kafka, conduit, avro, schema registry): hw-kafka
Lua: luardkafka
Node.js: node-rdkafka
OCaml: ocaml-kafka
PHP: phpkafka
PHP: php-rdkafka
Python: confluent-kafka-python
Python: PyKafka
Ruby: Hermann
Ruby: rdkafka-ruby
Rust: rust-rdkafka
Tcl: KafkaTcl
Swift: Perfect-Kafka

Users of librdkafka

kafkacat - Apache Kafka swiss army knife
Wikimedia's varnishkafka - Varnish cache web log producer
Wikimedia's kafkatee - Kafka multi consumer with filtering and fanout
rsyslog
syslog-ng
collectd
logkafka - Collect logs and send to Kafka
redBorder
Headweb
Produban's log2kafka - Web log producer
fuse_kafka - FUSE file system layer
node-kafkacat
OVH - AntiDDOS
otto.de's trackdrd - Varnish log reader
Microwish has a range of Kafka utilites for log aggregation, HDFS integration, etc.
aidp - kafka consumer embedded Lua scripting language in data process framework
Yandex ClickHouse
NXLog - Enterprise logging system, Kafka input/output plugin.
large unnamed financial institutions
and many more..
Let me know if you are using librdkafka

Usage

Requirements

The GNU toolchain
GNU make
pthreads
zlib-dev (optional, for gzip compression support)
libssl-dev (optional, for SSL and SASL SCRAM support)
libsasl2-dev (optional, for SASL GSSAPI support)
libzstd-dev (optional, for ZStd compression support)

NOTE: Static linking of ZStd (requires zstd >= 1.2.1) in the producer enables encoding the original size in the compression frame header, which will speed up the consumer. Use STATIC_LIB_zstd=/path/to/libzstd.a ./configure --enable-static to enable static ZStd linking. MacOSX example: STATIC_LIB_zstd=$(brew ls -v zstd | grep libzstd.a$) ./configure --enable-static

Instructions

Installing prebuilt packages

On Mac OSX, install librdkafka with homebrew:

$ brew install librdkafka

On Debian and Ubuntu, install librdkafka from the Confluent APT repositories, see instructions here and then install librdkafka:

$ apt install librdkafka-dev

On RedHat, CentOS, Fedora, install librdkafka from the Confluent YUM repositories, instructions here and then install librdkafka:

$ yum install librdkafka-devel

On Windows, reference librdkafka.redist NuGet package in your Visual Studio project.

For other platforms, follow the source building instructions below.

Building

  ./configure
  # Or, to automatically install dependencies using the system's package manager:
  # ./configure --install-deps
  # Or, build dependencies from source:
  # ./configure --install-deps --source-deps-only

  make
  sudo make install

NOTE: See README.win32 for instructions how to build on Windows with Microsoft Visual Studio.

NOTE: See CMake instructions for experimental CMake build (unsupported).

Usage in code

See examples/rdkafka_example.c for an example producer and consumer.

Link your program with -lrdkafka -lz -lpthread -lrt.

Documentation

The public APIs are documented in their respective header files:

The C API is documented in src/rdkafka.h
The C++ API is documented in src-cpp/rdkafkacpp.h

To generate Doxygen documents for the API, type:

make docs

Configuration properties are documented in CONFIGURATION.md

For a librdkafka introduction, see INTRODUCTION.md

Examples

See the examples/sub-directory.

Tests

See the tests/sub-directory.

Support

File bug reports, feature requests and questions using GitHub Issues

Questions and discussions are also welcome on irc.freenode.org, #apache-kafka, nickname Snaps.

Commercial support

Commercial support is available from Edenhill services

zhanlangorz/librdkafka