multiline codec ignores last line when defined in input section

[umarengasamy]

Please post all product and debugging questions on our forum. Your questions will reach our wider community members there, and if we confirm that there is a bug, then we can open a new issue here.

For all general issues, please provide the following details for fast resolution:

• Version: 7.2.0

• Operating System: Linux

• Config File (if you have sensitive info, please remove it):

• Sample Data: Java stack trace

• Steps to Reproduce:
  input {
  s3 {
  type => "multilinelogs"
  bucket => ""
  region => ""
  interval => "1"
  prefix => "xxx"
  delete => true
  role_session_name => ""
  sincedb_path => "/dev/null"
  additional_settings => {
  force_path_style => true
  follow_redirects => false
  }
  codec => multiline {
  pattern => "^\s"
  what => "previous"
  auto_flush_interval => 1
  }
  }

  }

[Philipp-Panzer]

I had a similar issue. Did you do any filtering on the s3 key that's usually part of every event's metadata?
Looking at the plugin's code, I think that the last line is actually flushed to the pipeline, but it's not initialized with the s3 metadata which might cause unexpected failures of filters. At least that's what happened to me. See #190 for the changes I made to solve my issue.

[umarengasamy]

I use s3 key , mutate and make it as fields. We have issue only when we use multiline codec.

[madmax76]

We are experiencing a similar strange behavior

%{[@metadata][s3][key]} is assigned to a field PATH, but during processing of the multiline data at some seemingly random points the metadata is lost and PATH becomes the string "%{[@metadata][s3][key]}"

input {
    s3 {
        access_key_id  => ""
        secret_access_key => ""
        bucket => ""
        region => ""
        prefix => ""
        exclude_pattern => "^((?!\.xml).)*$"
        codec => multiline {
            charset => "ISO8859-1"
            pattern => "<DataSection>"
            negate => true
            what => "previous"
        }
        additional_settings => {
            force_path_style => true
            follow_redirects => false
        }
    }
}

filter {
    # ADD PATH field
    mutate {
        add_field => {
            "PATH" => "%{[@metadata][s3][key]}"
        }
    }

[jgough]

This seems to be because the input plugin does not decorate events when flushing stateful codecs to the queue
https://github.com/logstash-plugins/logstash-input-s3/blob/master/lib/logstash/inputs/s3.rb#L242

[jgough]

Is this issue a dupe of #153 ?

[kaisecheng]

The issue is fixed in 3.8.3