Implement ECS-Compatibility Mode

[yaauie]

As a part of the effort to make plugins able to run in an ECS-Compatible manner
by default in an upcoming release of Logstash, this plugin needs to
implement an ECS-Compatibility mode that does not implicitly use
fields that conflict with ECS, including named captures in sub-patterns.

[yaauie]

@kares and I discussed a plan-of-attack here:

1. copy and modify the existing patterns that include implicit captures to instead capture to ECS-compliant fields (scope only includes targets of implicit captures, including nested patterns).
2. select either the legacy patterns or these new ECS-compliant patterns to load at plugin initialization, based on the value of the ecs_compatibility setting provided by the ECS Compatibility Support mixin.

[tomrade]

A quick fix while migrations / new patterns are being created would be recommend using something like the "target" option so legacy patterns can be used with ECS indices by just shoving all the grok fields under a nested folder ie "logtype.fieldname". I dont think this would need an extra option maybe just some docs. You wouldn't have any of the magic ECS correlation fields but you could ingest your logs for the moment without errors.

[kares]

Thanks Thomas, we already introduced target => ... in plugin version 4.3.0
and we are planning shipping both "legacy" patterns as well as those that will have their captures ecs compatible.

[kares]

Another think to consider will be defaulting overwrite => [ message ] in ECS mode. Since the source string is usually under message and most 'line' grok matches are going to produce a message field which is meant to replace the original field (which should be stored under event.original pretty much right after being constructed from the input).