Pretty printing in Logstash

[cyberluke]

I got parse issues with your extension, but stock log4j2 works with this config:

<JsonLayout compact="true" eventEol="true"/>

It inserts new line, I think you insert just a bunch of JSON nodes in the log file and he thinks it is an array, but there is no "," character.

My logstash config looks like this:
input {
file {
codec => json
type => "log4j-json"
path => "/logs/log4j2.log"
}
}

[cyberluke]

Does prettyPrint work for you? It adds a new line, which causes Logstash to throw exception about parse error. How do you configure it?

[vy]

@cyberluke, what does your layout+appender configuration look like? You need to provide a little bit more information so I can reproduce the issue. And yes, prettyPrintEnabled="true" works for me, though I am using log4j2-redis-appender.

[cyberluke]

I'm using standard file appender and logstash scans the log file as input (as seen in the first post). I tried also socket appender. Yeah, so I guess redis appender will do something different, so it can recognize individual lines correctly.

[vy]

Using the following setup

$ cat /tmp/logstash.conf 
input {
  file {
    codec => json
    type => "log4j-json"
    path => "/tmp/log4j2.log"
  }
}

output {
  stdout {}
}

$ echo -n > /tmp/log4j2.log

I have downloaded logstash-6.0.0 and run it as follows:

$ logstash-6.0.0/bin/logstash -f /tmp/logstash.conf

Then I ran com.vlkan.log4j2.logstash.layout.demo.LogstashLayoutDemo (shipped with log4j2-logstash-layout sources) from my IDE where I configure layout-demo/src/main/resources/log4j2.xml as follows:

<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="warn">
    <Appenders>
        <File name="FILE" fileName="/tmp/log4j2.log">
            <LogstashLayout dateTimeFormatPattern="yyyy-MM-dd'T'HH:mm:ss.SSSZZZ"
                            prettyPrintEnabled="false"
                            locationInfoEnabled="true"
                            templateUri="classpath:LogstashJsonEventLayoutV1.json"/>
        </File>
    </Appenders>
    <Loggers>
        <Root level="info">
            <AppenderRef ref="FILE"/>
        </Root>
    </Loggers>
</Configuration>

I was able to see Logstash swiftly displaying the polled log entries to console. Further, the log file content was JSON-per-line as expected:

$ cat /tmp/log4j2.log 
{"line_number":10,"class":"com.vlkan.log4j2.logstash.layout.demo.LogstashLayoutDemo","@version":1,"source_host":"varlik","message":"Hello, world!","thread_name":"main","@timestamp":"2017-12-04T19:49:10.207+01:00","level":"INFO","file":"LogstashLayoutDemo.java","method":"main","logger_name":"com.vlkan.log4j2.logstash.layout.demo.LogstashLayoutDemo"}
{"exception":{"exception_class":"java.lang.RuntimeException","exception_message":"test"},"line_number":12,"class":"com.vlkan.log4j2.logstash.layout.demo.LogstashLayoutDemo","@version":1,"source_host":"varlik","message":"Hello, error!","thread_name":"main","@timestamp":"2017-12-04T19:49:10.227+01:00","level":"ERROR","file":"LogstashLayoutDemo.java","method":"main","logger_name":"com.vlkan.log4j2.logstash.layout.demo.LogstashLayoutDemo"}

I am not a Logstash expert, but just happen to know Java to write my own JSON layout addressing certain shortcomings of other Log4j2 JSON layout plugins. If you can provide me a reproducible case (like I did above), I can help you with investigating the issue.

[cyberluke]

With prettyPrintEnabled="false" it worked before. I was just questioning how is it working for you with prettyPrintEnabled="true", because you mention this in your github readme :))

[vy]

Ah! Now I see your point. In the README, I do not mentioned any particular appender, AFAIK. The selling point of the layout plugin is the vast potential for customization. Though I cannot comment on whether all of these features will work with a particular appender or not. Maybe I should mention about this in a FAQ. Thanks for the tip.