- install
docker desktop
- install
docker-compose
pip install jmxquery plotly dash numpy
- download data from
https://www.kaggle.com/datasets/thoughtvector/customer-support-on-twitter/data
- put
twcs.csv
into./data
-
Kafka
- zookeeper
- brokers: kafka1, kafka2, kafka3
-
Hadoop
- namenode
- nodemanager
- resourcemanager
- historyserver
- datanodes: datanode1, datanode2, datanode3
-
Producer
-
Consumer
- consumers: consumer1, consumer2
launch docker containers for hadoop and kafka cluster
cd <YourPath>/Kafka-Pipeline
systemctl --user start docker-desktop
docker network create kafka-network
docker-compose build
chmod +x ./run-kafka-hadoop.sh
./run-kafka-hadoop.sh
in new terminal, run
chmod +x ./modify-env.sh
./modify-env.sh
to install python3 for all hadoop nodes, after all kafka and hadoop nodes are set, run
chmod +x ./run-producer-consumer.sh
./run-producer-consumer.sh
in new terminal to start the producer and consumers
start mapreduce job in new terminal
docker exec -it namdenode /bin/bash
# in the opened namenode terminal
cd /home/workspace/src/word_count
chmod +x ./run.sh
./run.sh
output should be in file /output/word_count/part-00000
of the hadoop hdfs
kafka-ui: localhost:8080
dash: localhost:8050
Hadoop HDFS interface: localhost:9870
Hadoop cluster interface: localhost:8088