This is a series of Jupyter Notebooks on how to start with Apache Kafka and Python. You can try these notebooks in order to learn the basic concepts of Kafka in an environment containing markdown text, media and executable code on the same page.
The notebooks are based on a managed Apache Kafka instance created on Aiven's website, but can be also customised to any Kafka instance running locally with SSL authentication. Aiven's offer 300$ of free credit that you can redeem by creating your account on Aiven's website.
If you have any question or improvement suggestion regarding the notebooks, please open an issue. Any contributions are welcome!
You can access the notebooks via Jupyterlab, this example will be based on docker
- clone the repository
- open a terminal
- go to the folder where the repository has been cloned
- run the following
docker run --rm -p 8888:8888 \
-e JUPYTER_ENABLE_LAB=yes \
-v "$PWD":/home/jovyan/work \
jupyter/datascience-notebook
You'll see a folder named work on the top left, under it you'll find the list of notebooks.
This repository contains the following notebooks.
- 00 - Aiven Setup.ipynb: Creates requires Apache Kafka and PostgreSQL instances on Aiven website
- 01 - Producer.ipynb: Creates a Python Producers and produces the first messages
- 02 - Consumer.ipynb: reads the messages produced by 01 - Producer.ipynb
- 03 - 00 - Partition Producer.ipynb: creates a topic with two partitions and pushes records in each partition
- 03 - 01 - Consumer - Partition 0.ipynb: reads messages pushed by 03 - 00 - Partition Producer.ipynb on partition 0
- 03 - 02 - Consumer - Partition 1.ipynb: reads messages pushed by 03 - 00 - Partition Producer.ipynb on partition 1
- 04 - New Consumer Group.ipynb: creates a new consumer part of a new consumer group and reads messages produced by 01 - Producer.ipynb
- 05 - Kafka Connect.ipynb: creates a new topic containing JSON messages with schema and payload, and then creates a Kafka Connect JDBC Connector to PostgreSQL
- ON - Aiven - Delete Services.ipynb: deletes all the instances created by 00 - Aiven Setup.ipynb
The notebooks are divided per Apache Kafka functionality.
Create Managed Apache Kafka and PostgreSQL instances with Aiven.io
00 - Aiven Setup.ipynb notebook downloads Aiven's command line interface and creates an Apache Kafka and a PostgreSQL instance.
Please change <INSERT_TOKEN_HERE> and <INSERT_EMAIL_HERE> with a valid email address and token created on Aiven's website. The notebook creates the instances and also stores all the required connection credentials locally.
01 - Producer.ipynb Creates a Python Kafka Producer and produces the first messages. After the first message is produced, open the 02 - Consumer.ipynb notebook and pace it alongside the Producer.
02 - Consumer.ipynb reads from the topic where 01 - Producer wrote. But it does it from the point in time that it attaches to Kafka, not going back to history.
If you want to read messages created with 01 - Producer you need to run 02 - Consumer.ipynb's last code block before producing any messages on 01 - Producer. This behaviour is Kafka's default and can be changed by adding a line 'auto.offset.reset'='earliest' to the consumer properties.
Partitions is Apache Kafka are a way to divide messages belonging to the same topic in sub-logs.
- 03 - 00 - Partition Producer.ipynb creates a topic with two partitions using
KafkaAdminand sends a message to each partition. We can then open both 03 - 01 - Consumer - Partition 0.ipynb and 03 - 02 - Consumer - Partition 1.ipynb which will read messages fromPartition 0andPartition 1respectively.
Messages in Apache Kafka are not deleted when read from a consumer. This makes them available for other consumers to be read. 04 - New Consumer Group.ipynb creates a new consumer part of the a new Consumer Group and reads from the topic where 01 - Producer wrote. We can check now, by sending a message from the 01 - Producer notebook, that we can receive it both in 02 - Consumer.ipynb and 04 - New Consumer Group.
Kafka Connect is a prebuilt framework enabling an easy integration of Apache Kafka with existing data sources or sinks. Aiven provides Kafka connect as managed service making the integration a matter of a single config file. 05 - Kafka Connect.ipynb: Creates a new Kafka topic containing messages with both schema and payload, and then pushes them to a PostgreSQL database via Kafka Connect.
Once you're done, you can delete all the services create on Aiven's website by executing the code in ON - Aiven - Delete Services.ipynb
We maintain some other resources that you may also find useful:
This project is licensed under the Apache License, Version 2.0.