This repository is to demonstrate a problem I have with logback and the logstash encoder. Under certain circumstances logmessages are lost.
To reproduce this behaviour I build this minimal example.
Please set the destination of the logstash appender in line 10 in logback-spring.xml to your logstash server.
Our input logstash server has a very simple input configuration:
input {
tcp {
port => 9000
codec => json_lines
}
}
Start a netcat listening on 8081 with
nc -Lp 8081Just start the application main class with your IDE.
- In the output from the application I see the log messages 1-4.
- In the output from netcat I see the log messages 1-4, too.
- In kibana, that is connected to the elasticsearch cluster behind the logstash server, I see only log messages 1 and 4.
- Why are log messages lost?
- Does it have something to do with the JSON string and not using
appendRaw? - Do I have to test the content of my message to decide, if I need
appendorappendRaw?
We found the problem. To help others, who come across this question, I will explain the problem and the solution here.
The log messages were successfully transmitted to logstash and elasticsearch. So there were no errors in those logs. The reason, that we could not find the log entries was the field for the "payload". We have multiple services and one of them was sending an object instead of a string. That lead to a mapping conflict and the log messages could not be accessed.
The solution for us was to define an explicit mapping and to introduce a naming convention. Now we have a text field payload and an object field payloadAsObject. This convention is used in all of our services.
The problematic log entries were far enough in the past, so we could delete the old index and solve the conflict. But there are possibilities to reindex the data without downtime. For example this one: https://medium.com/craftsmenltd/rebuild-elasticsearch-index-without-downtime-168363829ea4