logstash-plugins/logstash-filter-fingerprint

Support to exclude fields from `concatenate_all_fields`

Opened this issue · 0 comments

I would like to use all fields in an event for fingerprint calculation except a few of them which are non-deterministic when re-importing logs using the file or unix socket input for example. This is useful when the list of fields varies and is not fully known. Example use case:

fingerprint {
  concatenate_all_fields => true
  exclude => [
    "#logstash_timestamp",

    "@version",
    "path",
    "port",
    "input_type",
    "type",
    "source",

    "host",
    "@timestamp"
  ]
  target => "[@metadata][_id]"
  method => "SHA512"
  key => "XXX"
  base64encode => true
}

Hint: The @timestamp is excluded because when you re-import logs, Logstash sets the @timestamp at input stage and it is only later overwritten by the timestamp extracted from the log line. Fingerprint calculation is done as early as possible to still generate the same fingerprint even when the processing/transformation of the event is changed.