Fields not supported for ASN database
pemontto opened this issue · 2 comments
pemontto commented
Logstash crashes when supplying valid fields for the ASN database.
@filter=<LogStash::Filters::GeoIP id=>\"enrich: src asn\", cache_size=>10000, source=>\"src_ip\", target=>\"src_asn\", database=>\"GeoLite2-ASN.mmdb\", fields=>[\"as_org\", \"asn\"], enable_metric=>true, periodic_flush=>false, lru_cache_size=>1000>>", :error=>"illegal field value as_org. valid values are [AUTONOMOUS_SYSTEM_NUMBER, AUTONOMOUS_SYSTEM_ORGANIZATION, CITY_NAME, COUNTRY_NAME, CONTINENT_CODE, CONTINENT_NAME, COUNTRY_CODE2, COUNTRY_CODE3, IP, ISP, POSTAL_CODE, DMA_CODE, REGION_NAME, REGION_CODE, TIMEZONE, LOCATION, LATITUDE, LONGITUDE, ORGANIZATION]"}
- Version: Logstash 5.4.3, logstash-filter-geoip 4.2.1
- Operating System: Centos7
- Config File (if you have sensitive info, please remove it):
geoip {
id => "enrich: dest asn"
cache_size => 10000
source => "dest_ip"
target => "dest_asn"
database => "GeoLite2-ASN.mmdb"
fields => [ "as_org", "asn"]
tag_on_failure => []
}
- Sample Data:
{"src_ip": "2.8.8.8"}
I was expecting the code to allow any number of fields and do a best effort to extract them rather than have a static list which can get outdated.
omnidepp commented
I can confirm, that it behaves strangely with asn/isp database (getting the same error).
This worked for me, though: Specify the fields you want like this:
geoip {
source => "id.resp_h"
target => "resp_isp"
database => "/opt/geoip/GeoIP2-ISP.mmdb"
fields => ["ISP", "AUTONOMOUS_SYSTEM_NUMBER", "AUTONOMOUS_SYSTEM_ORGANIZATION", "ORGANIZATION"]
}
sliddjur commented
Can confirm this happens for me too on logstash-filter-geoip (6.0.3)
I am using elasticsearch 7.9.3, and I have a new (not legacy) index template called logstash-*
.
If I comment out this line: #fields => [ as_org, asn ]
in my geoip filter, the error goes away.
Nov 17 21: 58: 34 elastic-srv logstash[1442798]: [2020 - 11 - 17 T21: 58: 34, 475][ERROR][logstash.agent] Failed to execute action {
: id =>: main,: action_type => LogStash::ConvergeResult::FailedAction,: message => "Could not execute action: PipelineAction::Reload<main>, action_result: false",: backtrace => nil
}
Nov 17 21: 58: 34 elastic-srv logstash[1442798]: [2020 - 11 - 17 T21: 58: 34, 501][INFO][logstash.outputs.elasticsearch][main] Attempting to install template {
: manage_template => {
"index_patterns" => "logstash-*", "version" => 60001, "settings" => {
"index.refresh_interval" => "5s", "number_of_shards" => 1
}, "mappings" => {
"dynamic_templates" => [{
"message_field" => {
"path_match" => "message", "match_mapping_type" => "string", "mapping" => {
"type" => "text", "norms" => false
}
}
}, {
"string_fields" => {
"match" => "*", "match_mapping_type" => "string", "mapping" => {
"type" => "text", "norms" => false, "fields" => {
"keyword" => {
"type" => "keyword", "ignore_above" => 256
}
}
}
}
}], "properties" => {
"@timestamp" => {
"type" => "date"
}, "@version" => {
"type" => "keyword"
}, "geoip" => {
"dynamic" => true, "properties" => {
"ip" => {
"type" => "ip"
}, "location" => {
"type" => "geo_point"
}, "latitude" => {
"type" => "half_float"
}, "longitude" => {
"type" => "half_float"
}
}
}
}
}
}
}