GeoIP stopped working as before :confused:

Question

GeoIP stopped working as before :confused:

dusatvoj opened this issue 3 years ago · 17 comments

Logstash information:

Please include the following information:

Logstash version (e.g. bin/logstash --version) 7.12.1
Logstash installation source (e.g. built from source, with a package manager: DEB/RPM, expanded from tar or zip archive, docker) - DEB package from elastic repo
How is Logstash being run (e.g. as a service/service manager: systemd, upstart, etc. Via command line, docker/kubernetes) - systemd
How was the Logstash Plugin installed - bundled in DEB package

JVM (e.g. java -version): java 11.0.11

If the affected version of Logstash is 7.9 (or earlier), or if it is NOT using the bundled JDK or using the 'no-jdk' version in 7.10 (or higher), please provide the following information:

JVM version (java -version)
JVM installation source (e.g. from the Operating System's package manager, from source, etc).
Value of the JAVA_HOME environment variable if set.

... I'm using Debian repo's openjdk-11

OS version (uname -a if on a Unix-like system): Linux elk-node-1 4.19.0-16-amd64 #1 SMP Debian 4.19.181-1 (2021-03-19) x86_64 GNU/Linux

Description of the problem including expected versus actual behavior:
I have problem that cca 1/4 events from apache / nginx / haproxy have tag _geoip_lookup_failure (some of them are RFC 1918 IPs) but the rest of events not have that tag. Below is image of Index pattern data with data types. In past (I think ~3 months ago) there were different types of some fields (geoip.ip changed from ip to text and geoip.location changed from geo_point into object). This behavior causes Field type conflict ... Due to Field type conflict I can't load map layer and even on new index patterns (eg. I've recently added new haproxy into different space and pattern) I can't create map layers bcs of wrong field type for geoip mapping

My logstash setup (only these basic fields related to geoip plugin:

geoip {
    source => "event.clientip"
}

Expected behavior is "no change of datatype" and "working out of the box as before"

Steps to reproduce:

IDK ... maybe upgrade of ELK VM (including upgrade of ELK)? Can't rememeber (and I've found out this shit a week after it started happening)

Provide logs (if relevant):
N/A 😕

Answer 1 · 2021-05-24T09:32:59.000Z

I meet this problem too! Please resolve it，thanks!

es logs are as below:

[logstash-2021.05.24][0] mapping update rejected by primary
java.lang.IllegalArgumentException: mapper [geoip.ip] cannot be changed from type [ip] to [text]
        at org.elasticsearch.index.mapper.FieldMapper.checkIncomingMergeType(FieldMapper.java:281) ~[elasticsearch-7.12.1.jar:7.12.1]
        at org.elasticsearch.index.mapper.FieldMapper.merge(FieldMapper.java:266) ~[elasticsearch-7.12.1.jar:7.12.1]
        at org.elasticsearch.index.mapper.FieldMapper.merge(FieldMapper.java:46) ~[elasticsearch-7.12.1.jar:7.12.1]
        at org.elasticsearch.index.mapper.ObjectMapper.doMerge(ObjectMapper.java:516) ~[elasticsearch-7.12.1.jar:7.12.1]
        at org.elasticsearch.index.mapper.ObjectMapper.merge(ObjectMapper.java:476) ~[elasticsearch-7.12.1.jar:7.12.1]
        at org.elasticsearch.index.mapper.ObjectMapper.doMerge(ObjectMapper.java:503) ~[elasticsearch-7.12.1.jar:7.12.1]
        at org.elasticsearch.index.mapper.RootObjectMapper.doMerge(RootObjectMapper.java:309) ~[elasticsearch-7.12.1.jar:7.12.1]
        at org.elasticsearch.index.mapper.ObjectMapper.merge(ObjectMapper.java:476) ~[elasticsearch-7.12.1.jar:7.12.1]
        at org.elasticsearch.index.mapper.RootObjectMapper.merge(RootObjectMapper.java:304) ~[elasticsearch-7.12.1.jar:7.12.1]
        at org.elasticsearch.index.mapper.Mapping.merge(Mapping.java:107) ~[elasticsearch-7.12.1.jar:7.12.1]
        at org.elasticsearch.index.mapper.MapperService.mergeMappings(MapperService.java:523) ~[elasticsearch-7.12.1.jar:7.12.1]
        at org.elasticsearch.index.mapper.MapperService.mergeMappings(MapperService.java:513) ~[elasticsearch-7.12.1.jar:7.12.1]
        at org.elasticsearch.index.mapper.MapperService.mergeAndApplyMappings(MapperService.java:389) ~[elasticsearch-7.12.1.jar:7.12.1]
        at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:384) ~[elasticsearch-7.12.1.jar:7.12.1]
        at org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:281) [elasticsearch-7.12.1.jar:7.12.1]
        at org.elasticsearch.action.bulk.TransportShardBulkAction$2.doRun(TransportShardBulkAction.java:164) [elasticsearch-7.12.1.jar:7.12.1]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [elasticsearch-7.12.1.jar:7.12.1]
        at org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:209) [elasticsearch-7.12.1.jar:7.12.1]
        at org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:115) [elasticsearch-7.12.1.jar:7.12.1]
        at org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:74) [elasticsearch-7.12.1.jar:7.12.1]
        at org.elasticsearch.action.support.replication.TransportWriteAction$1.doRun(TransportWriteAction.java:168) [elasticsearch-7.12.1.jar:7.12.1]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:732) [elasticsearch-7.12.1.jar:7.12.1]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [elasticsearch-7.12.1.jar:7.12.1]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
        at java.lang.Thread.run(Thread.java:831) [?:?]

Answer 2 · 2021-05-24T09:48:05.000Z

What version of the geoip are you using? (you can find with bin/logstash-plugin list --verbose geoip)

Answer 3 · 2021-05-24T09:51:01.000Z

@jsvd The version which is bundled with Logstash 7.12.1 DEB package ... logstash-filter-geoip (6.0.5)

Answer 4 · 2021-05-24T09:52:49.000Z

But It's happening about 3 months as I said above. I've created this issues when I found out that nobody cares (so maybe nobody knows about the potential bug or smth is wrong in my deployement)

Answer 5 · 2021-05-24T10:55:39.000Z

I'm trying to replicate on my side. Here's a few tests:

Index the same document with a geoip lookup with Logstash 7.9.0, then with Logstash 7.12.1. Using Elasticsearch 7.12.1

Logstash command used for both:

echo "1.1.1.1" | bin/logstash -e "filter { geoip { source => message  } } output { elasticsearch {} }"

No indexing issues, mapping of the "geoip.ip" field remained type "ip":

❯ curl -s localhost:9200/logstash-*/_mapping | jq ".\"logstash-2021.05.24-000001\".mappings.properties.geoip.properties.ip"
{
  "type": "ip"
}

✅ template correct, both documents indexed

Send failed lookup message first, then a valid lookup document:

echo "111111" | bin/logstash -e "filter { geoip { source => message  } } output { elasticsearch {} }"

then

echo "1.1.1.1" | bin/logstash -e "filter { geoip { source => message  } } output { elasticsearch {} }"

✅ template correct, both documents indexed

From there I can't replicate the template change of client.ip from "ip" to "text". is there sample data or documents that can be shared here? Are you able to replicate with a few curl requests or logstash configurations?

Answer 6 · 2021-05-24T12:35:09.000Z

The version of this plugin is also 6.0.5

logstash configuration is as blew:

input {
    redis {
        data_type   => "list"
        key         => "logstash-log-list"
        host        => "127.0.0.1"
    }
}
filter {
    mutate {
        lowercase => [ "type" ]
    }
    if [clientPosition][cip] {
        geoip {
            source => "[clientPosition][cip]"
            database => "/usr/local/logstash/GeoLite2-City_20210507/GeoLite2-City.mmdb"
        }
    }
    if [timestamp] {
        date {
            match => ["timestamp", "yyyy-MM-dd HH:mm:ss", "yyyy-M-d HH:mm:ss", "ISO8601", "UNIX"]
        }
        mutate {
            remove_field => [ "timestamp" ]
        }
    }
    if ![type] {
        drop{}
    }
    if [type] =~ "test" {
        mutate { add_field => { "[@metadata][target_index]" => "logstash-%{type}-%{+YYYY.MM}" } }
    } else if [type] =~ "^dap" {
        mutate { add_field => { "[@metadata][target_index]" => "logstash-%{type}-%{+YYYY.MM.dd}" } }
    } else {
        mutate { add_field => { "[@metadata][target_index]" => "logstash-%{type}-%{+YYYY.MM}" } }
    }
}
output {
    stdout {
        codec => "rubydebug"
    }
    elasticsearch {
        hosts              => ["https://127.0.0.1:9200"]
        cacert             => "/usr/local/logstash/config/elasticsearch-ca.pem"
        index              => "%{[@metadata][target_index]}"
        document_type      => "%{type}"
        ilm_enabled        => false
        sniffing           => true
    }
}

Test data

127.0.0.1:6379> LPUSH logstash-log-list '{"type": "log", "clientPosition":{"cip":"223.101.75.206"}}'

Error comes again

Could not load '.aprc' from ENV['HOME']: no implicit conversion of nil into String
{
              "type" => "log",
    "clientPosition" => {
        "cip" => "223.101.75.206"
    },
             "geoip" => {
              "latitude" => 31.2449,
                    "ip" => "223.101.75.206",
          "country_name" => "China",
         "country_code2" => "CN",
              "location" => {
            "lon" => 121.3948,
            "lat" => 31.2449
        },
              "timezone" => "Asia/Shanghai",
         "country_code3" => "CN",
           "region_name" => "Shanghai",
             "city_name" => "Putuo",
             "longitude" => 121.3948,
           "region_code" => "SH",
        "continent_code" => "AS"
    },
        "@timestamp" => 2021-05-24T12:32:08.785Z,
          "@version" => "1"
}
[2021-05-24T12:32:09,493][WARN ][logstash.outputs.elasticsearch][main][6aceaa243065e491686cfa938155a077bb1ee954222efab248a34080ff331a5b] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"logstash-log-2021.05", :routing=>nil, :_type=>"log"}, #<LogStash::Event:0x54ab69a7>], :response=>{"index"=>{"_index"=>"logstash-log-2021.05", "_type"=>"log", "_id"=>"4rJcnnkB8VlYSZD9cuLF", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"mapper [@version] cannot be changed from type [keyword] to [text]"}}}}

Check template is ok

curl -k -s -XGET  'https://127.0.0.1:9200/logstash-*/_mapping'  | jq ".\"logstash-log-2021.05\".mappings.properties.geoip.properties.ip"
{
  "type": "ip"
}

Template full

{
    "mappings" : {
      "dynamic_templates" : [
        {
          "message_field" : {
            "path_match" : "message",
            "match_mapping_type" : "string",
            "mapping" : {
              "norms" : false,
              "type" : "text"
            }
          }
        },
        {
          "string_fields" : {
            "match" : "*",
            "match_mapping_type" : "string",
            "mapping" : {
              "fields" : {
                "keyword" : {
                  "ignore_above" : 256,
                  "type" : "keyword"
                }
              },
              "norms" : false,
              "type" : "text"
            }
          }
        }
      ],
      "properties" : {
        "@timestamp" : {
          "type" : "date"
        },
        "@version" : {
          "type" : "keyword"
        },
        "geoip" : {
          "dynamic" : "true",
          "properties" : {
            "ip" : {
              "type" : "ip"
            },
            "latitude" : {
              "type" : "half_float"
            },
            "location" : {
              "type" : "geo_point"
            },
            "longitude" : {
              "type" : "half_float"
            }
          }
        }
      }
    }
  }

Answer 7 · 2021-05-24T13:13:36.000Z

This may be related to a change in the ES output related to the handling of the document_type logstash-plugins/logstash-output-elasticsearch#994

Can y'all confirm that you're explicitly setting document_type => value is the ES output? I can see that @hectorqin is

Answer 8 · 2021-05-24T13:21:35.000Z

After i remove the document_type setting, it works, thanks!

Answer 9 · 2021-05-24T16:04:22.000Z

I do not have explicitly set document_type :/

elasticsearch {
    hosts => "https://127.0.0.1:9200"
    ssl => "true"
    cacert => "<CERT_PATH>"
    ssl_certificate_verification => "false"
    index => "logstash-%{[app][name]}-%{[app][env]}-%{+YYYY.MM.dd}"
    ilm_policy => "<JUST_HOT_TO_WARM_PHASE>"
    user => "XXXXXXXXXX"
    password => "XXXXXXXXXX"
}

I have trouble trying inline logstash filter bcs SSL and auth so I haven't tried that. If it's necessary I can try that but I need a bit of help with multiple options in inline filter definition.

Answer 10 · 2021-05-24T18:04:10.000Z

The elasticsearch output plugin gives mi this mapping (copied from kibana) ... no document_type set:

{
  "geoip": {
    "properties": {
      "city_name": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "continent_code": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "country_code2": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "country_code3": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "country_name": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "dma_code": {
        "type": "long"
      },
      "ip": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "latitude": {
        "type": "float"
      },
      "location": {
        "properties": {
          "lat": {
            "type": "float"
          },
          "lon": {
            "type": "float"
          }
        }
      },
      "longitude": {
        "type": "float"
      },
      "postal_code": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "region_code": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "region_name": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "timezone": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      }
    }
  }
}

Answer 11 · 2021-05-25T10:30:32.000Z

@dusatvoj in ES 7.x the type won't be included in the response unless ?include_type_name=true is included in the request.
For example:

❯ curl -s "localhost:9200/_mapping?pretty" | head -n5                     
{
  "logstash-2021.05.24-000001" : {
    "mappings" : {
      "dynamic_templates" : [
        {

❯ curl -s "localhost:9200/_mapping?include_type_name=true&pretty" | head -n5
{
  "logstash-2021.05.24-000001" : {
    "mappings" : {
      "_doc" : {
        "dynamic_templates" : [

Answer 12 · 2021-05-25T12:03:05.000Z

So you mean output like this? https://pastebin.com/nmhpxdFX

curl --insecure --user X:Y -XGET "https://localhost:9200/logstash-moo-hosting-all-2021.05.25/_mapping?include_type_name=true&pretty"

Answer 13 · 2021-05-25T12:57:20.000Z

Thank you can you also obtain the template with the type? e.g.

curl --insecure --user X:Y -XGET -s https://localhost:9200/_template/logstash?include_type_name=true

Answer 14 · 2021-05-25T21:05:39.000Z

@jsvd https://localhost:9200/_template/logstash?include_type_name=true&pretty:

{
  "logstash" : {
    "order" : 0,
    "version" : 60001,
    "index_patterns" : [
      "logstash-*"
    ],
    "settings" : {
      "index" : {
        "number_of_shards" : "1",
        "refresh_interval" : "5s"
      }
    },
    "mappings" : {
      "_doc" : {
        "dynamic_templates" : [
          {
            "message_field" : {
              "path_match" : "message",
              "mapping" : {
                "norms" : false,
                "type" : "text"
              },
              "match_mapping_type" : "string"
            }
          },
          {
            "string_fields" : {
              "mapping" : {
                "norms" : false,
                "type" : "text",
                "fields" : {
                  "keyword" : {
                    "ignore_above" : 256,
                    "type" : "keyword"
                  }
                }
              },
              "match_mapping_type" : "string",
              "match" : "*"
            }
          }
        ],
        "properties" : {
          "@timestamp" : {
            "type" : "date"
          },
          "geoip" : {
            "dynamic" : true,
            "properties" : {
              "ip" : {
                "type" : "ip"
              },
              "latitude" : {
                "type" : "half_float"
              },
              "location" : {
                "type" : "geo_point"
              },
              "longitude" : {
                "type" : "half_float"
              }
            }
          },
          "@version" : {
            "type" : "keyword"
          }
        }
      }
    },
    "aliases" : { }
  }
}

Answer 15 · 2021-11-04T14:21:51.000Z

@dusatvoj This issue will be transferred to elasticsearch-output as the problem is not related to geoip-filter. geoip-filter gives location details of the IP address and keeps the output geoip.ip as a string of ip. In your case, the field type in elasticsearch changed. It could be an issue of elasticsearch upgrade or elasticsearch template.

Answer 16 · 2021-11-04T14:27:22.000Z

@kaisecheng ok, thank you for info

Answer 17 · 2021-11-04T14:50:06.000Z

new issue created logstash-plugins/logstash-output-elasticsearch#1050