Race condition between register and finish_register lands data in ecs_logstash instead of data stream

[jsvd]

The register method has a sequence of actions:

1. setup_template_manager_defaults(data_stream_enabled)
2. after_successful_connection w/ finish_register
3. sets up @index according to data stream

During 2. a new thread is launched to execute finish_register once a connection is established.

A race condition can happen if the order is:

1. thread is launched in register to run finish_register
2. the first set of instructions from finish_register execute
3. the rest of register executes
4. ilm_setup executes.

This sequence causes install_template to be called before @index is set to a data stream here.
During install_template, ilm_in_use? is called, and ilm_actually_enabled becomes true.
Once finish_register gets to setup_ilm if ilm_in_use?, ilm_in_use? is true and so @index is overwritten here.

[jsvd]

The only available work around at this point is to disable data streams entirely and construct the data stream name through string interpolation (and also disabling template management and ILM):

output {
    elasticsearch {
        cloud_id => "..."
        cloud_auth => ".."
        data_stream => false
        # alternative to data_stream_auto_routing => true
        index => "%{[data_stream][type]}-%{[data_stream][dataset]}-%{[data_stream][namespace]}"
        # data streams only support create
        action => "create"
        manage_template => false
        ilm_enabled => false
    }
}