/event-driven-microservices

Primary LanguageTypeScript

Building event-driven systems with kafka

What are we building?

  • A simple event-driven system that will allow us to send messages to kafka and then consume them

Housekeeping

You will need

  1. Docker
  2. Node.js

Starting Kafka & Conductor

Follow the instructions in the conduktor folder

Install dependencies

yarn add kafkajs

What will you learn?

  • Setup a Kafka cluster for local development with Docker
  • Connect a producer to the cluster
  • Connect a consumer to the cluster
  • Graceful shutdown - disconnect the producer and consumer from the cluster
  • Using Conduktor to view & manage the cluster

Why Kafka?

  • Kafka is a distributed streaming platform
  • It is horizontally scalable
  • It is fault tolerant
  • It is fast
  • It is used by many companies including Uber, Netflix, LinkedIn, and many more

Kafka terms and definitions

  • Producer - A producer is a client that sends messages to a topic on the Kafka cluster

  • Consumer - A consumer is a client that reads messages from a topic on the Kafka cluster

  • Topic - A topic is a category or feed name to which records are published. Topics in Kafka are always multi-subscriber

  • Broker - A broker is a server that hosts a topic. Each broker is identified by a unique id

  • Partition - A partition is an ordered, immutable sequence of records that is continually appended to a file. The records in the partitions are each assigned a sequential id number called the offset that uniquely identifies each record within the partition.

  • Replication - Replication is the process of duplicating data across multiple brokers. Replication provides fault tolerance and high availability. A topic can have multiple partitions, and each partition can have multiple replicas.

  • Consumer Group - A consumer group is a group of consumers that cooperate to consume messages from a set of topics. Each consumer in a group can process the messages in parallel. The Kafka cluster ensures that each message is delivered to one consumer in each subscribing consumer group.

  • Offset - The offset is a unique identifier for a message within a partition. The offset is assigned when a producer publishes a message to a partition. The offset is used by the consumer to determine the message to process next.

  • Leader - The leader is the server that is currently the active controller for a partition. The leader handles all read and write requests for the partition. Only the leader can become a follower.

  • Follower - The follower is a passive server that replicates the log of the leader. The follower can become the leader if the current leader fails.