Determine if `data_streams => true` can be possible with ECS compatibility effectively disabled
yaauie opened this issue · 0 comments
Currently, we prohibit the use of data_stream => true
, and resolve the default data_stream => auto
to false
when ECS is effectively disabled. The primary driver of this is to prevent users from getting themselves into a situation implicitly that would cause them trouble.
We need to determine in what conditions a customer should be able enable data streams features when they also want to run the pipeline without ECS.
When running on Logstash 8, ECS is on-by-default, but that can be:
- overridden for an individual plugin by providing
ecs_compatibility => disabled
, - or the default itself can be changed by setting
pipeline.ecs_compatibility: disabled
for an individual pipeline or for the process as a whole.
When ECS is disabled, the behaviour in this plugin is changed in a couple of ways:
- when
manage_templates => true
,template
defaults to a pre-ecs template data_stream => auto
resolves tofalse
When data streams is effectively enabled, manage_templates
defaults to false
.
Therefore, when a user explicitly asks for data_stream => true
, and does not also set manage_templates => true
, the effective value of ecs_compatibility
should have no effect. We need to verify this assertion, and if it holds true, should begin allowing users who explicitly ask for data streams to operate with or without ECS enabled.