Can't rename @metadata fields

[pemontto]

Mutate doesn't rename nested @metadata fields correctly.

Config

• Copy contents of @metadata to meta
• Rename [@metadata][one] to [@metadata][two]
• Copy contents of @metadata to meta2

input {
  stdin {add_field => {"[@metadata][one]" => "test1"}}
}

filter {
  mutate {
    replace => [ "meta", "%{@metadata}" ]
  }
  mutate {
    rename => [ "[@metadata][one]", "[@metadata][two]" ]
  }
  mutate {
    replace => [ "meta2", "%{@metadata}" ]
  }
}

output {
  stdout { codec => rubydebug }
}

Result

• [@metadata][one] still exists with its original value
• [@metadata][two] exists and is set to null

Logstash startup completed
abcd
{
       "message" => "abcd",
      "@version" => "1",
    "@timestamp" => "2016-06-16T04:48:15.395Z",
          "meta" => "{\"one\":\"test1\"}",
         "meta2" => "{\"one\":\"test1\",\"two\":null}"
}

Result with non-metadata field

• Same config just replace @metadata with test

Logstash startup completed
abcd
{
       "message" => "abcd",
      "@version" => "1",
    "@timestamp" => "2016-06-16T04:49:28.239Z",
          "test" => {
        "two" => "test1"
    },
          "meta" => "{\"one\":\"test1\"}",
         "meta2" => "{\"two\":\"test1\"}"
}

[pemontto]

Tested the latest version, issue still occurs. Showing a different config

input {
  stdin { codec => json }
}
filter {
  mutate {
    replace => [ "[@metadata][type1]", "%{metadata_type1}" ]
    remove_field => [ "@metadata_type1" ]
  }
  mutate {
    rename => [ "@metadata_type2", "[@metadata][type2]" ]
  }
  mutate {
    add_field =>  { 
      "met1" => "%{[@metadata][type1]}"
      "met2" => "%{[@metadata][type2]}"
    }
  }
}

output {
  stdout { codec => rubydebug }
}

{"metadata_type1": "test1", "metadata_type2": "test2"}
12:52:47.171 [[main]>worker7] DEBUG logstash.pipeline - filter received {"event"=>{"@timestamp"=>2016-11-23T12:52:46.530Z, "metadata_type1"=>"test1", "@version"=>"1", "host"=>"dev-logstoreb2.e-secure.com.au", "metadata_type2"=>"test2", "tags"=>[]}}
12:52:47.172 [[main]>worker7] DEBUG logstash.filters.mutate - filters/LogStash::Filters::Mutate: removing field {:field=>"@metadata_type1"}
12:52:47.174 [[main]>worker7] DEBUG logstash.util.decorators - filters/LogStash::Filters::Mutate: adding value to field {"field"=>"met1", "value"=>["%{[@metadata][type1]}"]}
12:52:47.174 [[main]>worker7] DEBUG logstash.util.decorators - filters/LogStash::Filters::Mutate: adding value to field {"field"=>"met2", "value"=>["%{[@metadata][type2]}"]}
12:52:47.175 [[main]>worker7] DEBUG logstash.pipeline - output received {"event"=>{"@timestamp"=>2016-11-23T12:52:46.530Z, "met2"=>"%{[@metadata][type2]}", "metadata_type1"=>"test1", "met1"=>"test1", "@version"=>"1", "host"=>"dev-logstoreb2.e-secure.com.au", "metadata_type2"=>"test2", "tags"=>[]}}
{
        "@timestamp" => 2016-11-23T12:52:46.530Z,
              "met2" => "%{[@metadata][type2]}",
    "metadata_type1" => "test1",
              "met1" => "test1",
          "@version" => "1",
              "host" => "dev-logstoreb2.e-secure.com.au",
    "metadata_type2" => "test2",
              "tags" => []
}

[pemontto]

Checked on Logstash 5.5, still an issue

[berglh]

This persists in Logstash 6.3.2

[geekofalltrades]

We're experiencing this in Logstash 6.3.1, as well.

[TheVastyDeep]

The problem here is this line of code. 'event.remove("[@metadata][a]")' does not return its value or remove it. It appears to be a no-op. That's a core event API issue. Not sure where to check if there is an issue open for it.

[jsvd]

confirmed:

/tmp/logstash-6.7.0 % bin/logstash -i irb
irb(main):001:0> e = LogStash::Event.new("[@metadata][field]" => "1", "field" => 2)
=> #<LogStash::Event:0x4236d55>
irb(main):002:0> e.remove("field")
=> 2
irb(main):003:0> e.remove("[@metadata][field]")
=> nil

[edit]

this also happens with any nested structure:

irb(main):001:0> e = LogStash::Event.new("[hey][you]" => 1)
=> #<LogStash::Event:0x234a21e9>
irb(main):002:0> e.remove("[hey][you]")
=> nil

[TheVastyDeep]

Not a problem with other nested structures

irb(main):001:0> e = LogStash::Event.new()
=> #<LogStash::Event:0x5ced169a>
irb(main):002:0> e.set("[hey][you]", 1)
=> 1
irb(main):003:0> e.remove("[hey][you]")
=> 1
irb(main):004:0> e.remove("[hey][you]")
=> nil

[msb-dev]

Still seeing the same problem with logstash 7.1.0. Interestingly, if the source @metadata field has the same name as a root-level field, the root field's contents is renamed instead of the @metadata field:

input {
  stdin {add_field => {
    "[one]" => "test1ROOT"
    "[@metadata][one]" => "test1"
  }}
}

filter {
  mutate {
    replace => [ "meta1", "%{@metadata}" ]
  }
  mutate {
    rename => [ "[@metadata][one]", "[@metadata][two]" ]
  }
  mutate {
    replace => [ "meta2", "%{@metadata}" ]
  }
}

output {
  stdout { codec => rubydebug }
}

• [@metadata][one] still exists
• [@metadata][two] exists and is set to test1ROOT
• [one] no longer exists

I see similar behaviour when renaming from a @metadata field to a root level field (i.e. replace [@metadata][two] with two everywhere in the above snippet).

[jgregmac]

Still present in version 7.4.2. This is very unexpected behavior, and worse, Logstash returns no errors to indicate that it effectively is tossing out critical data. Please fix this. It caused me to waste a good deal of time over the past two days.

[colinsurprenant]

remove_field not working against @metadata is reported in elastic/logstash#4124

[colinsurprenant]

Thanks all for the issue reports. Fixed per elastic/logstash#11334, closing.