Filestream not fully implemented

[Fabian1976]

We found out that after upgrading to the most recent filebeat puppet module our filebeat harvesters for JSON files (ndjson) didn't work anymore.

The JSON filtering when using filestream as an input_type also changed.

The work-around for us was to explicitly add input_type: log to the config because the default input_type was changed to filestream (which should have been a major release instead of a minor according to semver).

When the input_type is set to log, the JSON block in the filebeat config looks like this:

  json:
      message_key: 'log'
      keys_under_root: true
      overwrite_keys: true

But when filestream is used, the JSON block should look like this:

  parsers:
    - ndjson:
        message_key: 'log'
        keys_under_root: true
        overwrite_keys: true

(according to these docs)

But the old JSON block is still used when the input_type is set to filestream.

Could you please fix this?

[psychonaut]

Moreover, changing from input_type:log to input_type: filestream without take_over: true re-ingested from the beginning all log files which could be a huge problem (it was for us).

https://www.elastic.co/guide/en/beats/filebeat/current/_step_2_enable_the_take_over_mode.html

[stevenpost]

@pcfens Would you consider re-opening this issue? It seems this was marked as completed with merging #326, but that is a completely separate issue.

[antaflos]

I have created PR #341 to add support for the take_over setting that helps prevent Filebeat from re-ingesting already seen log entries when switching from log to filestream.