/log-aggregator

Log aggregation pipeline with kafka and ELK stack

Primary LanguagePython

log-aggregator

Log aggregation pipeline with Kafka and ELK stack

Data Pipeline Architecture

Data pipeline

Kibana Dashboard

Log data visualization:

Kibana dashboard

Run Project

  • Set up .env variables
  • Run docker run --build to build and run the containers
  • Open localhost:5601 to view the Kibana dashboard
  • Create data views for the logs.

Note: You might need to increase your Docker memory allocation to 4GB to run all containers efficiently.