Fields are lost when sent to sumologic

[xdays]

here's our config:

input {
kinesis {
kinesis_stream_name => "test"
region => "us-west-2"
initial_position_in_stream => "LATEST"
codec => json { }
}
}

filter {
if [service] != "naboo" {
drop { }
}
if [service] == "naboo" {
drop {
percentage => 90
}
}
mutate {
add_field => { "env" => "prod" }
rename => ["instance_id", "host"]
}
}

output {
sumologic {
url => "https://endpoint6.collection.us2.sumologic.com/receiver/v1/http/test
cookies => false
}
stdout {}
}

the raw log from our stdout output plugin is:

{
"service" => "test",
"private_ip" => "172.31.16.82",
"@timestamp" => 2020-05-01T08:27:13.068Z,
"message" => "2020-04-24 08:26:23.207 request_id=Fgi0AJLilk66pVUATSpi [info] Sent 200 in 2ms",
"env" => "prod",
"@version" => "1",
"host" => "i-05ttt69ab21126c13e"
}

But we can't see service, host and private_ip fields from sumologic query.

[himanshub16]

This plugin tries to format the events in a specified format, the default format being %{@timestamp} %{host} %{message}, which doesn't send other fields to the output/Sumo Logic.
example config

You might want to modify your configuration as

output {
  sumologic {
    url => "https://endpoint6.collection.us2.sumologic.com/receiver/v1/http/test
      format => "%{@json}"
  }
  stdout {}
}

I was able to ingest from a local file with following config.
The screenshot shows the difference due to format.

input {
  file {
    path => "/root/flog.log"
      start_position => "beginning"
  }
}


filter {
  mutate {
    add_field => { "env" => "prod" }
    rename => ["instance_id", "host"]
  }
}

output {
  sumologic {
    url => "redacted"
   # url => "https://enk2k6tgiof1a.x.pipedream.net"
      format => "%{@json}"   ## this makes all the difference
      compress => true
      compress_encoding => "gzip"
      interval => 1 # batch message up to 10s
      pile_max => 1024000 # batch message up to 1MB
      queue_max => 409600 # keep 400K
      sender_max => 100
  }
  stdout {}
}

[himanshub16]

Alternatively, Sumo also provides support to apply fields via HTTP header X-Sumo-Fields.

This will require following changes

• Specifying extra_headers in config
  https://github.com/SumoLogic/logstash-output-sumologic/#parameters-of-plugin

output {
  sumologic {
    url => "https://endpoint6.collection.us2.sumologic.com/receiver/v1/http/test
      extra_headers
  }
  stdout {}
}

• Enabling fields from Sumo
  https://help.sumologic.com/Manage/Fields#manage-fields

For this particular issue, I guess adding format should be enough.

[xdays]

Great, thanks!