Support additional json encoders/formatters
nadworny opened this issue · 17 comments
I'm trying to implement ECS formatting and haven't found a way to do it using quarkus. ECS is becoming a standard format for many enterprises and I was wondering how difficult it would be to implement it in your plugin?
I was able to format the ExtLogRecord using the ECS library like this but this doesn't replace the whole log entry, it adds a new field:
import co.elastic.logging.jboss.logmanager.EcsFormatter;
import io.quarkiverse.loggingjson.JsonProvider;
import io.quarkiverse.loggingjson.JsonGenerator;
import org.jboss.logmanager.ExtLogRecord;
@Singleton
public class MyJsonProvider implements JsonProvider {
@Override
public void writeTo(JsonGenerator generator, ExtLogRecord event) throws IOException {
var ecsFormatter = new EcsFormatter();
generator.writeStringField("testingxxx", ecsFormatter.format(event));
}
}Just add this dependency:
<dependency>
<groupId>co.elastic.logging</groupId>
<artifactId>jboss-logmanager-ecs-formatter</artifactId>
<version>1.1.0</version>
</dependency>Any suggestions?
hello,
I had same problem... my solution was simply to disable all all other fields:
Hope it helps!
Thanks but what I need is to replace the whole log statement with the ECS formatter. At the moment I'm only able to add the ECS formatted string as a new field which isn't going to be properly ingested by Elastic.
yes, I had exactly same constraint!
You can disable everything and you just get a starting {
So if I understood you correctly, when I disable all the fields and write a new field using generator, it should wrap it as a main json? I did the following but it still writes it as a new field and on top of that is also escaping the " in the strings. Here is how it looks:
{"":"{\"@timestamp\":\"2021-09-27T07:43:08.945Z\", \"log.level\": \"INFO\", \"message\":\"Live reload total time: 0.636s \", \"ecs.version\": \"1.2.0\",\"process.thread.name\":\"Timer-0\",\"log.logger\":\"io.quarkus.deployment.dev.RuntimeUpdatesProcessor\"}\n"}Code:
@Override
public void writeTo(JsonGenerator generator, ExtLogRecord event) throws IOException {
var ecsFormatter = new EcsFormatter();
generator.writeStringField("", ecsFormatter.format(event));
}Could you please provide a code snippet of how you set it up?
I cannot share my code (it is a custom formatter for another "unfied log format" we have internally in SIX).
But of course you can't do that:
generator.writeStringField("", ecsFormatter.format(event));
ecsFormatter is giving you a JSON as a string... so you should first parse it back and then "copy" all the content in the generator.
Otherwise, you could propose a PR in the ECS repository to extend quarkus logging directly.
I think I got you. But how do I "copy" all the content in the generator? Which method do you use? You don't have to paste here the whole code, just the one that is invoking the generator interface.
well, you have several options:
- parse and copy all keys to the generator. Pseudo-code:
Json json = Json.parse(ecsFormatter.format(event));
writeToGenerator(generator, key, json)
def writeToGenerator(generator, key, value) {
if (value instanceof JsonNumber) {
generator.writeNumber(key, value);
} else if (value instanceof JsonString) {
generator.writeString(key, value);
} else if (value instanceof JsonObject) {
for (key in keys(value)) {
writeToGenerator(generator, key, value.get(key))
}
} // TODO: handle arrays too
it could be a bit tricky since it is a recursive algorithm. And it is not very efficient too (since you have to parse JSON and recreate it with another shape).
- reimplement the ECS format directly using the generator. You could use
EcsJsonFormatteras reference:
in both cases you have to disable all providers (like per my original comment).
- Another option would be to write a custom quarkus extension that uses directly the
EcsFormatterto format the log. Basically providing another implementation ofExtFormatter, like this library:
Hope it helps :-)
PS
In my company we started with option 3. and then switched to option 2. few months back and we are quite happy now as we don't have to maintain a custom quarkus extension!
Thanks a lot fellow Zuricher @dfa1 ;)
This is what I came up with:
- Serializer: https://gist.github.com/nadworny/5fc96555c5f5d24c919fcacc482e820c
- Formatter: https://gist.github.com/nadworny/a80d405b4b4fc50f6e078ca14f451ab4
Everything seems to be working properly except of the following:
public static void serializeOrigin(JsonGenerator generator, String fileName, String methodName, int lineNumber) throws IOException {
StringBuilder builderFileName = new StringBuilder();
StringBuilder builderFunction = new StringBuilder();
JsonUtils.quoteAsString(fileName, builderFileName);
JsonUtils.quoteAsString(methodName, builderFunction);
generator.writeStartObject();
var json = Json.createObjectBuilder()
.add("log", Json.createObjectBuilder()
.add("origin", Json.createObjectBuilder()
.add("file", Json.createObjectBuilder()
.add("name", builderFileName.toString())
.add("line", lineNumber))
.add("function", builderFunction.toString()))).build();
generator.writeObject(json);
generator.writeEndObject();
}I'm getting this error: https://gist.github.com/nadworny/3fd4a8a79fd0c7e8112cfcc1aabaf786
Any idea what am I doing wrong?
@nadworny you're welcome fellow Zuricher ;)
According to the stacktrace:
java.lang.RuntimeException: com.fasterxml.jackson.core.JsonGenerationException: Can not start an object, expecting field name (context: Object)
at io.quarkiverse.loggingjson.JsonFormatter.format(JsonFormatter.java:37)
there is an unexpected JSON object. What is on line 37? Maybe this line?
generator.writeObject(json);
if so, just use:
generator.writeObjectFieldStart("log"); // will write "log": {
generator.writeObjectFieldStart("origin"); // will write "origin": {
generator.writeObjectFieldStart("file"); // will write "file": {
NB
with the generator you can skip "quoting as string" as the generator is safe
Thanks, I'll try it tomorrow.
You mentioned once:
Otherwise, you could propose a PR in the ECS repository to extend quarkus logging directly.
Which repository did you mean? Which part of my code do you think I could propose?
It works as suggested, see the updated serializer class.
Awesome, I'll test it this week
hey @SlyngDK, it works, thanks a lot!
The only thing I'm missing but it's not a must atm are the log.origin and ecs.version fields.
I have added ecs.version in 1.1.1.
I am not sure about the log.origin part.
Take a look at the serializeOrigin https://gist.github.com/nadworny/5fc96555c5f5d24c919fcacc482e820c#file-ecsjsonquarkusserializer-java-L104 but I'm not sure if it's also part of 1.1.1 version? We are using 1.2.0...