/Spring-Boot-MS-ELK

To integrate ELK (Elasticsearch, Logstash, and Kibana) with a Spring Boot application, you can follow these steps. The goal is to set up logging with Elasticsearch for storing logs, Logstash for processing logs, and Kibana for visualization.

1. Elasticsearch Setup

First, ensure that Elasticsearch is installed and running on your machine or server. You can use Docker to run Elasticsearch:

docker run -d --name elasticsearch -p 9200:9200 -e "discovery.type=single-node" elasticsearch:7.10.0

2. Logstash Setup

Logstash is used to collect, process, and forward logs to Elasticsearch. Create a Logstash configuration file (logstash.conf) to define how the logs should be processed. Here's a basic example of a logstash.conf:

input {
  beats {
    port => 5044
  }
}

filter {
  json {
    source => "message"
  }
}

output {
  elasticsearch {
    hosts => ["http://localhost:9200"]
    index => "springboot-logs-%{+YYYY.MM.dd}"
  }
  stdout { codec => rubydebug }
}

Run Logstash with this configuration:

docker run -d --name logstash -p 5044:5044 -v /path/to/logstash.conf:/usr/share/logstash/pipeline/logstash.conf logstash:7.10.0

3. Kibana Setup

Kibana is used for visualizing logs stored in Elasticsearch. You can also run Kibana using Docker:

docker run -d --name kibana -p 5601:5601 --link elasticsearch:kibana elasticsearch:7.10.0

4. Spring Boot Application Setup

In your Spring Boot application, add the required dependencies to pom.xml (if you're using Maven):

<dependency>
    <groupId>net.logstash.logback</groupId>
    <artifactId>logstash-logback-encoder</artifactId>
    <version>6.6</version>
</dependency>

5. Logback Configuration

To send logs to Logstash, create or modify your logback-spring.xml file in the src/main/resources directory:

<configuration>
    <appender name="logstash" class="net.logstash.logback.appender.LogstashTcpSocketAppender">
        <destination>localhost:5044</destination>
        <encoder class="net.logstash.logback.encoder.LogstashEncoder" />
    </appender>

    <root level="INFO">
        <appender-ref ref="logstash" />
    </root>
</configuration>

This configuration will direct all logs to Logstash, which in turn forwards them to Elasticsearch.

6. Application Properties

Configure your application.properties or application.yml to fine-tune logging:

# Log Level Configuration
logging.level.root=INFO
logging.level.com.yourpackage=DEBUG

7. Testing and Visualization

After the configuration:

  1. Run your Spring Boot application.
  2. Logstash will receive the logs and forward them to Elasticsearch.
  3. Open Kibana by navigating to http://localhost:5601 in your browser.
  4. Create an index pattern for springboot-logs-* to visualize the logs.

8. Advanced Logging

You can further enhance logging by:

  • Using MDC (Mapped Diagnostic Context) for contextual logging.
  • Configuring different appenders for different environments (e.g., file-based logging in development).
  • Setting up custom fields to include more application-specific details.

Conclusion

This setup allows you to have a robust logging system using ELK with Spring Boot. It gives visibility into your application's behavior, performance, and errors, making it easier to monitor and debug. Let me know if you want more details on a specific part or an example of any advanced configuration!