logstash-plugins/logstash-filter-fingerprint

The result of Float type is not unique

Closed this issue · 2 comments

medcl commented

as reported from here:https://discuss.elastic.co/t/logstash-fingerprint/42327

The filter doesn't work if the data type is float,and the recreation is below:

1.create a test file /tmp/test_file with content

{"b":{"b1":1.1}}

2.use this logstash config

input {
file {
path => ["/tmp/test_file"]
sincedb_path => "/tmp/sincedb"
codec => json
type => "nginx_visit"
start_position => "beginning"

}
}

filter {
fingerprint {
method => "SHA1"
key => "160129"
source => ["[b]"]
}
}

output {
stdout { codec => json }
}

3.the result output

{"b":{"b1":1.1},"@version":"1","@timestamp":"2016-03-08T04:03:24.575Z","path":"/tmp/test_file","host":"medcls-MacBook.local","type":"nginx_visit","fingerprint":"00e4536734914ce8beb46480d14601ed473fb849"}

{"b":{"b1":1.1},"@version":"1","@timestamp":"2016-03-08T04:03:33.600Z","path":"/tmp/test_file","host":"medcls-MacBook.local","type":"nginx_visit","fingerprint":"04373a842b7b13daafd4519c3e5e430082acdc43"}

Tried both pipelines below, cannot reproduce.

Pipeline:

input {
  generator {
    lines => [
      '{"b":{"b1":1.1}}',
      '{"b":{"b1":1.1}}'
    ]
    count => 1
    codec => json
  }
}

filter {
  fingerprint {
    method => "SHA1"
    key => "160129"
    source => ["[b]"]
  }
}

output {
  stdout {
    codec => json
  }
}

Output:

{"fingerprint":"5598acd401c93680b57020d49dfd89323b3b0a3a","host":"PC","@version":"1","@timestamp":"2018-10-12T22:19:43.505Z","b":{"b1":1.1},"sequence":0}{"fingerprint":"5598acd401c93680b57020d49dfd89323b3b0a3a","host":"PC","@version":"1","@timestamp":"2018-10-12T22:19:43.487Z","b":{"b1":1.1},"sequence":0}

Also tested:

{"b":{"b1":1.1}}
{"b":{"b1":1.1}}

Pipeline:

input {
  file {
    path => "./logstash-6.4.2/test"
    sincedb_path => './logstash-6.4.2/sincedb'
    codec => json
    start_position => "beginning"
  }
}

filter {
  fingerprint {
    method => SHA1
    key => '160129'
    source => [ '[b]' ]
  }
}

output {
  stdout {
    codec => json
  }
#  stdout {
#    codec => 'rubydebug'
#  }
}

Output:

{"@timestamp"=>2018-10-13T14:20:34.990Z, "fingerprint"=>"5598acd401c93680b57020d49dfd89323b3b0a3a", "@version"=>"1", "b"=>{"b1"=>1.1}, "host"=>"LUCA-PC", "path"=>"----"}
{"@timestamp":"2018-10-13T14:20:34.990Z","fingerprint":"5598acd401c93680b57020d49dfd89323b3b0a3a","@version":"1","b":{"b1":1.1},"host":"LUCA-PC","path":"----"}
medcl commented

maybe fixed, closing now.