ericchou1/devnetcreate2021-kafka-real-time-data-streaming-for-network-engineers

DevNet Create 2021 Talk: Kafka Real Time Data Streaming for Network Engineers

DevNet Create 2021 Talk: Kafka Real Time Data Streaming for Network Engineers

DevNet Create 2021 Talk: Kafka Real Time Data Streaming for Network Engineers

This repository is provided as supplement for Cisco DevNet Create 2021 Talk

Useful links:

• Kafka Introduction
• Kafka Documentation
• Engineering Kafka for Secure Autonomous Operations
• Cisco's Real-time Ingestion Architecture with Kafka and Druid

Full Playlist

• Clic image below for YouTube Playlist

Example 1: Create Topic and Test

In this example, we will use the Kafka console commands to produce and consume messages.

• Click image below for YouTube video

Example 2: Consumer Group and Key-Value Pair in Records

In this example, we will have multiple consumers in a consumer group as well as produce messages with key-value pairs.

The key is important for offset sequence and which partition the data will live in. Basically when key=null, data is sent round robin, if the key is set, they will be sent to the same partition and order be guaranteed. The key can be a user, a device, a database id, etc.

• Click image below for YouTube video

Example 3: Topic Offsets

In this example, we will take a closer look at the concept of offsets. The offsets are kept a per-topic, per-consumer group level.

• Click image below for YouTube video

Example 4: Simple Python Library Example

In this example, we will use the confluent-kafka Python library to produce and consume messages.

• Click image below for YouTube video