/kafka-lab

Primary LanguageJava

Kafka Course

Welcome

First, welcome to this course on Kafka.

Although Kafka is quite simple to install, we decided to make base it on Docker and Docker-Compose. This gives us a couple of advantages:

  • Easier installation:
    • As long as you can get Docker to run, we know that the Kafka installation will work
    • Install docker, then simply run the docker-compose file
  • Consistency between Windows, Mac and Linux
  • The ability to scale up and down the Kafka cluster

Link to the labs

This link will lead you to all the labs and examples

Outline

DAY 1

  • Introduction (Lecture ~ 20 min)
    • Who are we?
    • What is Kafka?
    • Explain first lab
  • Verify that everything is installed and working (Lab ~ 20 min)
    • Install Kafka through Docker
    • Run a simple example of Kafka
  • Introduction to Kafka (Lecture ~ 30 min)
    • Kafka under the hood
    • What is a topic?
    • What is a partition?
    • What is a producer?
    • What is a consumer?
  • Creating a topic and pass a message (Lab ~30 min)
    • Create a topic
    • Run a simple consumer
    • Run a simple producer
  • Dissecting the first example (Lecture/Discussion ~ 30 min)
    • Walk-through of the first lab
    • Question and answers
  • Design of Kafka topics and partitions (Lecture ~ 30 min)
    • Case study
    • How to select topics?
    • How to select partitions?
  • Exercise: Designing topics and partitions (Group Project ~ 20 min)
    • Design topics and partitions

DAY 2

  • Evaluation of the designs and suggested solutions (Discussion ~20 min)
    • Discussion of the suggested solution(s)
    • Recommended design of case study
  • Implement Topics and Partitions for case study (Lab ~30 min)
    • Define a topic and partition in Kafka
    • Create a consumer and producer
    • Run a test script
  • Scaling Kafka (Lecture ~30 min)
    • Kafka Brokers
    • Kafka Clusters
    • Cluster mirroring
    • Consumer groups
  • Streaming APIs for Kafka (Lecture ~20 min)
    • What is streaming?
    • Why use streams?
    • Programming to streams
    • Example streams using Spark
  • Streaming and IoT Case Study (Lab ~30 min)
    • Consume a stream from Kafka
    • Build a Spark application over the Kafka stream
  • Kafka Administration and Integration (Lecture ~30 Min)
    • Integration with Big Data tools (Storm, Spark, Hadoop)
    • Kafka Connect
    • Certified Kafka connectors
    • Kafka administration
    • Kafka monitoring
    • Security