Unclear description of logic that happens on an error in the "parser filter"
jwreford99 opened this issue · 0 comments
Describe the bug
emit_invalid_record_to_error in the parser filter is a setting which allows users to decide on the outcome that happens when a record is unable to parsed for one of three reasons.
If set to true it says that the record will be given the @ERROR label, and "If you want to ignore these errors, set false".
However, I was seeing different behaviour, and looking at the relevant piece of source code https://github.com/fluent/fluentd/blob/c48ca04a593f2f4628c545d74c90f6caba513bee/lib/fluent/plugin/filter_parser.rb#L72-L95
it became clear that the documentation isn't quite clear.
Notably, the source code clearly has an extra condition that if reserve_data is true then it will not discard these errors (as I would of expected), but instead it allows them to pass but with an unparsed record. I fully understand why this behaviour exists, but I think the documentation around this could be included.
Link to the problematic documentation
https://docs.fluentd.org/filter/parser#emit_invalid_record_to_error
https://docs.fluentd.org/filter/parser#reserve_data
Expected explanation
I would expect the documentation for emit_invalid_record_to_error to clearly say that if reserve_data is set to true, then the record will not be discarded but will instead be passed through without any parsing happening.
Additional context
An example configuration which was not clear from the documentation is:
<filter gearset.*>
@type parser
<parse>
@type json
json_parser json
</parse>
key_name log
reserve_data true
emit_invalid_record_to_error false
</filter>
I am happy to open a PR myself, but given this is my first open-source contribution I figured I would put an issue in.