A demo application implementing the Kappa architecture, based on Kafka, Flink, and ELK.
Project was initialized using the Maven archetype flink-quickstart-java.
./mvnw package
cd docker
docker-compose up
In one terminal:
docker exec -u 0 -it docker_kafka_1 bash
/opt/kafka/bin/kafka-topics.sh --create --zookeeper zookeeper:2181 --replication-factor 1 --partitions 1 --topic wiki-results
/opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server kafka:9092 --topic wiki-results --from-beginning
In another terminal:
docker exec -u 0 -it docker_jobmanager_1 bash
flink run -c io.github.ouyi.WikipediaAnalysis /code/target/kafka-flink-demo-1.0-SNAPSHOT.jar
Watch the updates in the first terminal, and visit http://localhost:8081 with a browser to play with the Flink UI.
To stop the execution, run in the second terminal:
flink list
flink cancel <jobId>
In one terminal, start the standalone local filesystem connector:
docker exec -u 0 -it docker_kafka_1 bash
/opt/kafka/bin/kafka-topics.sh --create --zookeeper zookeeper:2181 --replication-factor 1 --partitions 1 --topic test_input
/opt/kafka/bin/connect-standalone.sh /opt/kafka/config/connect-standalone.properties /opt/kafka/config/connect-file-source.properties
Start io.github.ouyi.KafkaFlinkSqlDemo
in IDE with the argument localhost:19092
.
In another terminal:
docker exec -u 0 -it docker_kafka_1 bash
cat >> /var/tmp/test_input.json
// Paste in test records from demo/src/main/resources/test_input.json
Create some visualizations using the index pattern kafka
with the Kibana Web UI available at http://localhost:5601, or alternatively use the following command1:
curl -XPOST localhost:5601/api/kibana/dashboards/import -H 'kbn-xsrf:true' -H 'Content-type:application/json' -d @./docker/kibana/config/dashboard_demo.json
Stop io.github.ouyi.KafkaFlinkSqlDemo
in IDE and run in yet another terminal:
docker exec -u 0 -it docker_jobmanager_1 bash
flink run -c io.github.ouyi.KafkaFlinkSqlDemo /code/target/kafka-flink-demo-1.0-SNAPSHOT.jar
Start io.github.ouyi.kafka.SoundVolumeProducer
in IDE.
Visit http://localhost:8081 with a browser to play with the Flink UI.
cd docker
docker-compose down -v
1: To export a dashboard with all its dependencies: curl -XGET localhost:5601/api/kibana/dashboards/export?dashboard=dashboard_id_hash > dashboard_demo.json
↩