missing ECS fields
emalzer opened this issue · 4 comments
Logstash information:
- Logstash version
# /usr/share/logstash/bin/logstash --version
Using bundled JDK: /usr/share/logstash/jdk
logstash 7.15.1
-
Logstash installation source (e.g. built from source, with a package manager: DEB/RPM, expanded from tar or zip archive, docker)
DEB / Ubuntu 18.04 -
How is Logstash being run (e.g. as a service/service manager: systemd, upstart, etc. Via command line, docker/kubernetes)
systemd -
How was the Logstash Plugin installed
core plugin
JVM (e.g. java -version
):
OpenJDK Runtime Environment (build 11.0.11+9-Ubuntu-0ubuntu2.18.04)
OpenJDK 64-Bit Server VM (build 11.0.11+9-Ubuntu-0ubuntu2.18.04, mixed mode, sharing)
OS version (uname -a
if on a Unix-like system):
Linux minion-em-elk01 4.15.0-162-generic #170-Ubuntu SMP Mon Oct 18 11:38:05 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Description of the problem including expected versus actual behaviour:
The current ECS scheme (1.12) uses the field geo.region_iso_code field, the geoip filter plugin actually uses the geo.region_code field, which does not comply with the ECS scheme. Also it inserts the country_code2 and country_code3 (which is fine for me :) ), but not the geo.country_iso_code.
Steps to reproduce:
root@minion-em-elk01:/usr/share/logstash# ./bin/logstash -e 'input { stdin { codec => json } } filter { geoip { source => "ip" target => "client" } } output { stdout {} }'
....
{ "ip": "52.168.174.229" }
{
"ip" => "52.168.174.229",
"@version" => "1",
"@timestamp" => 2021-12-06T09:32:26.081Z,
"host" => "minion-em-elk01",
"client" => {
"latitude" => 37.9254,
"location" => {
"lat" => 37.9254,
"lon" => -76.8591
},
"country_name" => "United States",
"dma_code" => 556,
"longitude" => -76.8591,
"postal_code" => "22560",
"country_code3" => "US",
"country_code2" => "US",
"region_code" => "VA",
"continent_code" => "NA",
"city_name" => "Tappahannock",
"ip" => "52.168.174.229",
"timezone" => "America/New_York",
"region_name" => "Virginia"
}
}
^C[WARN ] 2021-12-06 10:32:40.735 [SIGINT handler] runner - SIGINT received. Shutting down.
[INFO ] 2021-12-06 10:32:40.964 [[main]-pipeline-manager] javapipeline - Pipeline terminated {"pipeline.id"=>"main"}
[INFO ] 2021-12-06 10:32:41.931 [LogStash::Runner] runner - Logstash shut down.
@emalzer ECS is not enabled by default in Logstash 7.15
The event layout you shared is not ECS compatible.
To enable it, go to pipeline.yml
- pipeline.id: geoip_test
pipeline.ecs_compatibility: v1
config.string: input { stdin { codec => json } } filter { geoip { source => "ip" target => "client" } } output { stdout {} }
and start logstash.
@kaisecheng yes, you are right, I'm in the middle of mirgating to ECS and forgot the compatibility mode while trying to recreate the issue as simple as possible. I thus tried it again with the following configuration:
root@gs-dev00:/usr/share/logstash# ./bin/logstash -e 'input { stdin { codec => json } } filter { geoip { source => "ip" target => "client" } } output { stdout {} }' --pipeline.ecs_compatibility v1
...
{ "ip": "52.168.174.229" }
{
"ip" => "52.168.174.229",
"client" => {
"ip" => "52.168.174.229",
"mmdb" => {
"dma_code" => 556
},
"geo" => {
"region_name" => "Virginia",
"postal_code" => "22560",
"location" => {
"lon" => -76.8591,
"lat" => 37.9254
},
"city_name" => "Tappahannock",
"country_name" => "United States",
"timezone" => "America/New_York",
"continent_code" => "NA",
"region_code" => "VA",
"country_iso_code" => "US"
}
},
"@timestamp" => 2022-01-10T17:58:47.301Z,
"event" => {
"original" => "{ \"ip\": \"52.168.174.229\" }\n"
},
"host" => {
"hostname" => "gs-dev00"
},
"@version" => "1"
}
^C[WARN ] 2022-01-10 19:03:14.274 [SIGINT handler] runner - SIGINT received. Shutting down.
[INFO ] 2022-01-10 19:03:14.457 [[main]-pipeline-manager] javapipeline - Pipeline terminated {"pipeline.id"=>"main"}
[INFO ] 2022-01-10 19:03:15.388 [LogStash::Runner] runner - Logstash shut down.
I attached the --pipeline-compatibility v1
parameter. Then the fields are ECS compatible - except for the field region_code
which should be region_iso_code
.
@emalzer I see your point! Looking at the example in ECS schema, geo.region_iso_code
: CA-QC, geo.region_name
: Quebec, the region_code
that this plugin gives does not fit to any of them, but it leans toward region_iso_code
.
From the Maxmind database, the value belongs to iso_code
as well.
I will try to make it to region_iso_code
and seek feedback from others. Thank you for reporting the issue!
Fixed in #206