Kafka-Climates-Demo

A simple application to obtain the max temperature of the day and notify if the temperature is high. Producer to send the data from the API to a broker and obtain the data for analysis using a Consumer (Kafka).

Requirements

Docker

Note 1: I'm using Docker to create a Kafka Server (Kafka Broker), if you want to use without containerization you can follow this tutorial to set up the Kafka Server: https://kafka.apache.org/quickstart.
Note 2: To install Docker and Docker Compose, use the official documentation: https://docs.docker.com/engine/install/. I'm using Debian Bookworm in this application.

Running Kafka with Docker:

I'll use the docker compose file of the Conduktor.io that has several configurations, for this code we need the simplest configuration (single Zookeper and a single broker).
Clone the repository with the following command:

$ git clone https://github.com/conduktor/kafka-stack-docker-compose.git

Now, create a Kafka cluster with one Zookeeper and one Kafka broker

$ cd kafka-stack-docker-compose  # change to directory of docker compose
$ docker-compose -f zk-single-kafka-single.yml up -d

Check if both the services are running in the background (-d flag above)

$ docker-compose -f zk-single-kafka-single.yml ps

One way to interact with the server is by starting an interactive session on the Docker (kafka1 because it's the username configured):

docker exec -it kafka1 /bin/bash

Run the command to show a list of topics (if this is the first will be empty):

$ kafka-topics --list --bootstrap-server localhost:9092

Create a new topic using the command line called: “climate-events" with the following command:

kafka-topics --create --topic climate-events --bootstrap-server localhost:9092

Python

Create an environment using virtualenv and activate installing the two libraries required:

$ python -m venv .venv
$ source activate .venv/bin/activate # unix
$ pip install requests kafka-python

Overview of the API

The API is provided by Open-Meteo which is possible to obtain values of weather for different localities, see the documentation of the API: https://open-meteo.com/en/docs/climate-api.
The idea of the script run_climate_api.py is to obtain the value of the max temperature of the day from the start of the month until the date today. This will write on the broker on the topic "climate-events” created before.
We can see the events using a Consumer run_consumer_climate.py and check if the temperature is higher than a threshold will notify (a simple print in our program) that day the temperature is high.

Running

First, you can change the localization, modifying the latitude and longitude on the script run_climate_api.py (change the query). You can use the own site to obtain the values: https://open-meteo.com/en/docs

Second, open a new terminal to run the consumer run_consumer_climate.py and execute the following command:

$ python run_consumer_climate.py

Finally, open a second terminal and run the script to obtain the data from the API run_climate_api.py and when the value is higher than the threshold will on the terminal that is executing the consumer.

Stopping

For stop the python scripts, just close both terminals.
Stopping the docker composer (inside the directory with the script):

$ docker-compose -f zk-single-kafka-single.yml stop

Stopping and deleting all the resources change stop to down

$ docker-compose -f zk-single-kafka-single.yml down

BrenoAV/Kafka-Climate-Demo