If the input event contains nested JSON in the "message" field, it gets converted to a string

[andrei-kalach]

• Logstash version 8.0.0, built from source
• logstash-input-beats Plugin 6.1.5, built from source

We are ingesting JSON log files with FileBeat and are forwarding them to a LogStash Instance.

The FileBeat configuration is trivial:

filebeat.inputs:
- type: log
  enabled: true
  json.keys_under_root: true
  json.add_error_key: true
  paths:
    - C:\Samples\*

In LogStash a FileBeat input is configured with the default plain codec:

input { beats { port => 5044 } }

In JSON documents in log files the "message" field contains a nested JSON, example:

{"thread": "http-nio-127.0.0.1-10080-exec-1", "level": "INFO", "message": {"mykey": "panel2", "duration": 92, "user": "anonym"}, "loggerFqcn": "org.apache.logging.log4j.spi.AbstractLogger", "contextMap": {"APP": "Racy10", "BROWSER": "curl/7.58.0"},"timestamp": "2021-06-11T15:25:08.842+0200"}

We are expecting that the "message" will be available as a Hash in LogStash for filtering / transformation, but in reality it comes in LogStash as a string representation of a Ruby Hash and cannot be processed as JSON:
{"user"=>"anonym", "mykey"=>"panel2", "duration"=>92}

The cause seems to be here: https://github.com/logstash-plugins/logstash-input-beats/blob/master/lib/logstash/inputs/beats/message_listener.rb#L184

        hash.delete(FILEBEAT_LOG_LINE_FIELD).to_s

The target field ("message") is extracted here from the input event and converted to a string.

The problem can be easily reproduced with tests, in spec/inputs/beats/message_listener_spec.rb add following test and run it:

    context "when the message contains nested JSON" do
      let(:message) { MockMessage.new("abc", { "message" => Hash["a" => "test", "b" => 200], "@metadata" => {} }) }

      it "extract the event" do
        subject.onNewMessage(ctx, message)
        event = queue.pop
        expect(event.get("message")).to eq(Hash["a" => "test", "b" => 200])
      end
    end

The test fails than with the error:

bundle exec rspec spec/inputs/beats/message_listener_spec.rb

...

   ←[31m  expected: {"a"=>"test", "b"=>200}←[0m
     ←[31m       got: "{\"a\"=>\"test\", \"b\"=>200}"←[0m