/Kafka-for-computer-visions

This project store example using kafka in AI

Primary LanguagePython

How to run

In this project, I use kafka docker image.

Tutorial:

Youtube

Environment

  • Python=3.9.13
pip install -r requirements.txt
  • Install Java if not installed
sudo apt update
sudo apt install default-jre
java --version

Requirement

  • zookeeper -> To manage and run kafka
  • kafka
  • cmak -> To manage kafka broker, consumer, topic, ... In GUI

Install Zookeeper and Kafka by hand

Install Zookeeper and Kafa using docker

The version of Kafka here is: 2.3.0

docker-compose up -d

Install CMAK for kafka manager

  • Must run in linux
#clone repo
https://github.com/yahoo/CMAK.git

cd CMAK

./sbt clean dist

cd target/universal/

unzip cmak-xxxx.zip
  • Config the zookeeper host in cmak-xxxx/conf/application.conf

Change from cmak.zkhosts="kafka-manager-zookeeper:2181" to cmak.zkhosts="<your-ip-addr>:2181"

In my case i use ipconfig and see that my ip-addr is: 192.168.1.138 like in the config.py so my value will be cmak.zkhosts="192.168.1.138:2181"

  • Run app on port 8080:
cd cmak-xxxx 
bin/cmak -Dconfig.file=conf/application.conf -Dhttp.port=8080

Run 1 broker, 1 topic, 1 partition, 1 consumer

Producer

python test_send.py

Consumer

python test_receive.py

Run 1 broker, 1 topic, 2 partition, 2 consumer

Producer

python test_send.py

Consumer 1

python consumer1.py

Consumer 2

python consumer2.py