/librdkafka

The Apache Kafka C/C++ library

Primary LanguageCOtherNOASSERTION

librdkafka - the Apache Kafka C/C++ client library

Copyright (c) 2012-2018, Magnus Edenhill.

https://github.com/edenhill/librdkafka

Gitter chat Build status

librdkafka is a C library implementation of the Apache Kafka protocol, containing both Producer and Consumer support. It was designed with message delivery reliability and high performance in mind, current figures exceed 1 million msgs/second for the producer and 3 million msgs/second for the consumer.

librdkafka is licensed under the 2-clause BSD license.

For an introduction to the performance and usage of librdkafka, see INTRODUCTION.md

See the wiki for a FAQ.

NOTE: The master branch is actively developed, use latest release for production use.

Overview

  • High-level producer
  • High-level balanced KafkaConsumer (requires broker >= 0.9)
  • Simple (legacy) consumer
  • Compression: snappy, gzip, lz4, zstd
  • SSL support
  • SASL (GSSAPI/Kerberos/SSPI, PLAIN, SCRAM) support
  • Broker version support: >=0.8 (see Broker version compatibility)
  • Stable C & C++ APIs (ABI safety guaranteed for C)
  • Statistics metrics
  • Debian package: librdkafka1 and librdkafka-dev in Debian and Ubuntu
  • RPM package: librdkafka and librdkafka-devel
  • Gentoo package: dev-libs/librdkafka
  • Portable: runs on Linux, OSX, Win32, Solaris, FreeBSD, AIX, ...

Language bindings

Users of librdkafka

Usage

Requirements

The GNU toolchain
GNU make
pthreads
zlib-dev (optional, for gzip compression support)
libssl-dev (optional, for SSL and SASL SCRAM support)
libsasl2-dev (optional, for SASL GSSAPI support)
libzstd-dev (optional, for ZStd compression support)

NOTE: Static linking of ZStd (requires zstd >= 1.2.1) in the producer enables encoding the original size in the compression frame header, which will speed up the consumer. Use STATIC_LIB_zstd=/path/to/libzstd.a ./configure --enable-static to enable static ZStd linking. MacOSX example: STATIC_LIB_zstd=$(brew ls -v zstd | grep libzstd.a$) ./configure --enable-static

Instructions

Installing prebuilt packages

On Mac OSX, install librdkafka with homebrew:

$ brew install librdkafka

On Debian and Ubuntu, install librdkafka from the Confluent APT repositories, see instructions here and then install librdkafka:

$ apt install librdkafka-dev

On RedHat, CentOS, Fedora, install librdkafka from the Confluent YUM repositories, instructions here and then install librdkafka:

$ yum install librdkafka-devel

On Windows, reference librdkafka.redist NuGet package in your Visual Studio project.

For other platforms, follow the source building instructions below.

Building

  ./configure
  # Or, to automatically install dependencies using the system's package manager:
  # ./configure --install-deps
  # Or, build dependencies from source:
  # ./configure --install-deps --source-deps-only

  make
  sudo make install

NOTE: See README.win32 for instructions how to build on Windows with Microsoft Visual Studio.

NOTE: See CMake instructions for experimental CMake build (unsupported).

Usage in code

See examples/rdkafka_example.c for an example producer and consumer.

Link your program with -lrdkafka -lz -lpthread -lrt.

Documentation

The public APIs are documented in their respective header files:

To generate Doxygen documents for the API, type:

make docs

Configuration properties are documented in CONFIGURATION.md

For a librdkafka introduction, see INTRODUCTION.md

Examples

See the examples/sub-directory.

Tests

See the tests/sub-directory.

Support

File bug reports, feature requests and questions using GitHub Issues

Questions and discussions are also welcome on irc.freenode.org, #apache-kafka, nickname Snaps.

Commercial support

Commercial support is available from Edenhill services