Write MDC String values resembling a number as integer in JSON output
jug opened this issue · 3 comments
Problem:
We were using a GELF Logger for our Java services to log. Now we moved to a new Cloud platform which natively supports fluentbit, which does not support a GELF input but only log messages formatted in JSON. So we moved to using LogstashEncoder.
Another reason to get rid of gelf-logger is that it brings a lot of dependencies with it (that are needlessly increasing the need to track security CVEs).
A bit later we realized that the GELF logger has an auto discovery mode of MDC String values converting them into their JSON equivalent of a number (Long or Double). See https://logging.paluch.biz/mdc.html
With the switch to the LogstashEncoder this lead to problems for us with the MDC fields in the indexes of our ELK (Elastic Search cluster), which had numbers for certain fields allowing range searches in dashboards, e.g. HTTP status code for REST-API calls, but now ended there as text values.
I was searching around for other libraries or features addressing this issue, but didn't find anything useful.
Describe the solution you'd like
Instead we would prefer that the logstash-logback-encoder provides such a feature.
To illustrate what I have in mind, I created a pull request:
Describe alternatives you've considered
We could write our own MdcJsonProvider that does the above, but it depends on the inner workings of the lib and we have dozens of services where we need to integrate it. We would much prefer to have just configuration of the LogStashEncoder.
The other alternative we considered is to configure mappings in the ELK to do this conversion, but it would also require to adjust the dozens of indexes in multiple datacenter clusters, so would be a lot of work. Also likely with every new MDC field, another mapping or adding the new keys might need to be added everywhere, which makes this approach also very time consuming and error prone.