Field type of `geoip.ip` and `geoip.location` changed after ELK upgrade
kaisecheng opened this issue · 1 comments
This is an issuer transfer from logstash-plugins/logstash-filter-geoip#184
The field type of geoip.ip
changed from ip
to text
and geoip.location
changed from geo_point
to object
. This behavior causes Field type conflict ...
. The problem is suspected to relate to ELK upgrade to 7.12.1
Logstash information:
Please include the following information:
- Logstash version (e.g.
bin/logstash --version
) 7.12.1 - Logstash installation source (e.g. built from source, with a package manager: DEB/RPM, expanded from tar or zip archive, docker) - DEB package from elastic repo
- How is Logstash being run (e.g. as a service/service manager: systemd, upstart, etc. Via command line, docker/kubernetes) - systemd
- How was the Logstash Plugin installed - bundled in DEB package
JVM (e.g. java -version
): java 11.0.11
If the affected version of Logstash is 7.9 (or earlier), or if it is NOT using the bundled JDK or using the 'no-jdk' version in 7.10 (or higher), please provide the following information:
- JVM version (
java -version
) - JVM installation source (e.g. from the Operating System's package manager, from source, etc).
- Value of the
JAVA_HOME
environment variable if set.
... I'm using Debian repo's openjdk-11
OS version (uname -a
if on a Unix-like system): Linux elk-node-1 4.19.0-16-amd64 #1 SMP Debian 4.19.181-1 (2021-03-19) x86_64 GNU/Linux
Description of the problem including expected versus actual behavior:
I have problem that cca 1/4 events from apache / nginx / haproxy have tag _geoip_lookup_failure
(some of them are RFC 1918 IPs) but the rest of events not have that tag. Below is image of Index pattern data with data types. In past (I think ~3 months ago) there were different types of some fields (geoip.ip
changed from ip
to text
and geoip.location
changed from geo_point
into object
). This behavior causes Field type conflict
... Due to Field type conflict
I can't load map layer and even on new index patterns (eg. I've recently added new haproxy into different space and pattern) I can't create map layers bcs of wrong field type for geoip mapping
plugin setup
elasticsearch {
hosts => "https://127.0.0.1:9200"
ssl => "true"
cacert => "<CERT_PATH>"
ssl_certificate_verification => "false"
index => "logstash-%{[app][name]}-%{[app][env]}-%{+YYYY.MM.dd}"
ilm_policy => "<JUST_HOT_TO_WARM_PHASE>"
user => "XXXXXXXXXX"
password => "XXXXXXXXXX"
}
The index template copied from Kibana
{
"geoip": {
"properties": {
"city_name": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"continent_code": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"country_code2": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"country_code3": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"country_name": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"dma_code": {
"type": "long"
},
"ip": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"latitude": {
"type": "float"
},
"location": {
"properties": {
"lat": {
"type": "float"
},
"lon": {
"type": "float"
}
}
},
"longitude": {
"type": "float"
},
"postal_code": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"region_code": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"region_name": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"timezone": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
}
}
cc @dusatvoj
This is an ECS compatibility issue I believe. ECS is v8 by default, and [geoip] is not the target when ECS is enabled, so there is no reason for the ECS compatible template to make it a geo_point. You can always add your own template, or disable ECS using the ecs_compatibility option on the output.