logstash-plugins/logstash-output-elasticsearch

Determine if `data_streams => true` can be possible with ECS compatibility effectively disabled

yaauie opened this issue · 0 comments

Currently, we prohibit the use of data_stream => true, and resolve the default data_stream => auto to false when ECS is effectively disabled. The primary driver of this is to prevent users from getting themselves into a situation implicitly that would cause them trouble.

We need to determine in what conditions a customer should be able enable data streams features when they also want to run the pipeline without ECS.

When running on Logstash 8, ECS is on-by-default, but that can be:

  • overridden for an individual plugin by providing ecs_compatibility => disabled,
  • or the default itself can be changed by setting pipeline.ecs_compatibility: disabled for an individual pipeline or for the process as a whole.

When ECS is disabled, the behaviour in this plugin is changed in a couple of ways:

  • when manage_templates => true, template defaults to a pre-ecs template
  • data_stream => auto resolves to false

When data streams is effectively enabled, manage_templates defaults to false.

Therefore, when a user explicitly asks for data_stream => true, and does not also set manage_templates => true, the effective value of ecs_compatibility should have no effect. We need to verify this assertion, and if it holds true, should begin allowing users who explicitly ask for data streams to operate with or without ECS enabled.