logback-redis: A Java repository from roechi

#Logback Redis Batch Appender

Enables Java applications to log directly to redis via the jedis client as part of centralized logging with the ELK stack.

More specifically, it uses async appenders and JSON encoding of the logstash-logback-encoder project. Messages are sent to redis in batches for performance reasons. The redis sentinel functionality is supported.

Maven dependencies

<dependency>
   <groupId>de.idealo.logback</groupId>
	<artifactId>logback-redis</artifactId>
   <version>${version}</version>
</dependency>

Note: At the time of writing logback-redis is only available as source code. A binary release to the central repository is in preparation.

Configuration

Parameters

connectionConfig:
- key: key under which messages are stored in redis
- scheme (NODE | SENTINEL): defines whether redis is accessed via a single node or via sentinel
- for scheme=SENTINEL:
  - sentinelMasterName: name of the sentinel master
  - sentinels: comma separated list of sentinels with the following structure: host1:port1,host2:port2
- for scheme=NODE:
  - host: redis host
  - port: redis port
maxBatchMessages: number of messages which are sent as batch size to redis
maxBatchSeconds: time interval in seconds after a batch of messages is sent to redis if the batch size is not reached
encoder: encoder for JSON formatting of the messages
ringBuffer and waitStrategyType determine how the logstash-logback-encoder asynchronously processes the messages. Note that messages may be lost if the ring buffer size is too small ("If the RingBuffer is full (e.g. due to slow network, etc), then events will be dropped.").

Example Configuration

<?xml version="1.0" encoding="UTF-8"?>
<included>
    <shutdownHook class="ch.qos.logback.core.hook.DelayingShutdownHook"/>
    <appender name="REDIS_APPENDER" class="net.logstash.logback.appender.LoggingEventAsyncDisruptorAppender">
        <ringBufferSize>131072</ringBufferSize>
        <appender class="de.idealo.logback.appender.RedisBatchAppender">
            <connectionConfig>
                <!-- redis sentinel: -->
                <scheme>SENTINEL</scheme>
                <sentinelMasterName>mymaster</sentinelMasterName>
                <sentinels>server:26379</sentinels>
                <!-- redis node: -->
                <!--<scheme>NODE</scheme>-->
                <!--<host>server</host>-->
                <!--<port>6379</port>-->
                <key>keyForRedis</key>
            </connectionConfig>
            <maxBatchMessages>1000</maxBatchMessages>
            <maxBatchSeconds>10</maxBatchSeconds>
            <encoder class="net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder">
                <providers>
                    <mdc/>
                    <pattern>
                        <pattern>
                            {
                            "timestamp": "%d{yyyy-MM-dd'T'HH:mm:ss.SSSZZ}",
                            "message": "%message",
                            "logger": "%logger",
                            "thread": "%thread",
                            "level": "%level",
                            "host": "${HOSTNAME}",
                            "file": "%file",
                            "line": "%line",
                            "app": "${projectName}"
                            }
                        </pattern>
                    </pattern>
                    <stackTrace>
                        <throwableConverter class="net.logstash.logback.stacktrace.ShortenedThrowableConverter">
                            <maxDepthPerThrowable>30</maxDepthPerThrowable>
                            <maxLength>4096</maxLength>
                            <shortenedClassNameLength>20</shortenedClassNameLength>
                            <rootCauseFirst>true</rootCauseFirst>
                        </throwableConverter>
                   </stackTrace>
                </providers>
            </encoder>
        </appender>
    </appender>
</included>

This appender configuration can either be included in a logback.xml file (via "included" tag) or be directly contained in a logback.xml (without "included" tag).

JSON Format Created by the Appender (Example) (= Input for Logstash)

{
    "_index": "myIndex-2015.09.15.12",
    "_type": "logs",
    "_id": "AU_RAKiIuARjD1TqcEFe",
    "_score": null,
    "_source": {
        "mdcKey1": "value1",
        "mdcKey2": "value2",
        "seq": "198",
        "timestamp": "2015-09-15T14:35:19.256+0200",
        "message": "logback-1:198",
        "logger": "LoggingTest",
        "thread": "main",
        "level": "INFO",
        "host": "myHost",
        "file": "?",
        "line": "?",
        "@version": "1",
        "@timestamp": "2015-09-15T12:35:25.251Z",
        "type": "logs"
},
    "fields": {
    "@timestamp": [
        1442320525251
    ]
},
    "sort": [
    1442320525251
]
}

Logging Appender Errors to a File

The following logger configuration in the logback.xml is recommended in order to write error messages of the appender directly to a file in error situations (especially if redis is not available):

<logger name="de.idealo.logback.appender" level="error" additivity="false">
    <appender-ref ref="FILE_FOR_REDISBATCHAPPENDER"/>
</logger>

Shutdown

Shutdown Hook

The redis batch appender must be shut down on application shutdown in order to ensure that cleans up background threads and pools and ensures that remaining messages are sent to Redis before shutting down the app. This is performed by the stop method of the redis batch appender that is automatically called when putting a shutdown hook in logback.xml:

<shutdownHook class="ch.qos.logback.core.hook.DelayingShutdownHook"/>

Spring Boot Apps

The shutdown hook above doesn't work in Spring Boot apps when they are shut down via the actuator shutdown URL (POST {baseUrl}/shutdown). Instead, this can be done by the following Spring component:

@Component
public class LogbackStopListener implements ApplicationListener<ContextClosedEvent> {
    @Override
    public void onApplicationEvent(final ContextClosedEvent event) {
        LogbackUtils.stopLogback();
    }
}

roechi/logback-redis