winweb/springboot-elk-prometheus-grafana

The goal of this project is to implement a Spring Boot application, called movies-api, and use Filebeat & ELK Stack (Elasticsearch, Logstash and Kibana) to collect and visualize application's logs and Prometheus & Grafana to monitor application's metrics.

springboot-elk-prometheus-grafana

The goal of this project is to implement a Spring Boot application, called movies-api, and use Filebeat & ELK Stack (Elasticsearch, Logstash and Kibana) to collect and visualize application's logs and Prometheus & Grafana to monitor application's metrics.

Note: In kubernetes-minikube-environment repository, it's shown how to deploy this project in Kubernetes (Minikube)

Application

• movies-api

  Spring Boot Web Java application that exposes a REST API for managing movies. Its endpoints are displayed in the picture below.

  

Prerequisites

• Java 11+
• Docker
• Docker-Compose

Start Environment

• Open a terminal and inside springboot-elk-prometheus-grafana root folder run

  docker-compose up -d
  

• Wait for Docker containers to be up and running. To check it, run

  docker-compose ps
  

Running application with Maven

• Open a terminal and make sure you are inside springboot-elk-prometheus-grafana folder

• Run the following command

  ./mvnw clean spring-boot:run --projects movies-api
  

  Note: If you want to change to "non-json-logs" (maybe during development it's useful), run

  ./mvnw clean spring-boot:run --projects movies-api -Dspring-boot.run.jvmArguments="-Dspring.profiles.active=non-json-logs"
  

Running application as Docker container

• Build Docker image

  • In a terminal, make sure you are inside springboot-elk-prometheus-grafana root folder
  • Run the following script to build the image
    • JVM

      ./docker-build.sh
      

    • Native (it's not implemented yet)

      ./docker-build.sh native
      

• Environment variables

  Environment Variable	Description
  MYSQL_HOST	Specify host of the MySQL database to use (default localhost)
  MYSQL_PORT	Specify port of the MySQL database to use (default 3306)

• Start Docker container

  • In a terminal, run the following command to start the Docker container

    docker run --rm --name movies-api -p 8080:8080 -e MYSQL_HOST=mysql \
      --network=springboot-elk-prometheus-grafana_default \
      ivanfranchin/movies-api:1.0.0
    

    Note: If you want to change to "non-json-logs", add -e SPRING_PROFILES_ACTIVE=non-json-logs to the command above

Application & Services URLs

• movies-api

  movies-api Swagger is http://localhost:8080/swagger-ui.html

• Prometheus

  Prometheus can be accessed at http://localhost:9090

  

• Grafana

  Grafana can be accessed at http://localhost:3000

  • In order to login, type admin for both username and password
  • You can skip the next screen that ask you to provide a new password
  • Click General / Home on the top
  • Click movies-api-dashboard

  

• Kibana

  Kibana can be accessed at http://localhost:5601

  IMPORTANT: in order to see movies-api logs in Kibana, you must run the application as Docker container

  Configuration

  • Access Kibana website
  • Click Explore on my own
  • On the main page, click the "burger" menu icon and, then click Discover
  • Click Create index pattern button
  • In the Name field, set filebeat-*
  • In the Timestamp field combo-box, select @timestamp
  • Click Create index pattern button
  • Click the "burger" menu icon again and then click Discover to start performing searches

  

• Elasticsearch

  Elasticsearch URL is http://localhost:9200

  Useful queries

  # Check ES is up and running
  curl localhost:9200
  
  # Check indexes in ES
  curl "localhost:9200/_cat/indices?v"
  
  # Check filebeat index mapping
  curl "localhost:9200/filebeat-*/_mapping"
  
  # Simple search
  curl "localhost:9200/filebeat-*/_search?pretty"
  

Shutdown

• To stop application, go to the terminal where it is running and press Ctrl+C
• To stop and remove docker-compose containers, network and volumes, go to a terminal and, inside springboot-elk-prometheus-grafana root folder, run the following command

  docker-compose down -v
  

Cleanup

To remove the Docker images created by this project, go to a terminal and, inside springboot-elk-prometheus-grafana root folder, run the script below

./remove-docker-images.sh

References

https://medium.com/@sece.cosmin/docker-logs-with-elastic-stack-elk-filebeat-50e2b20a27c6