GeoLite2 support

Question

GeoLite2 support

jordansissel opened this issue 9 years ago · 34 comments

(This issue was originally filed by @coder3000 at elastic/logstash#1709)

Will logstash support geolite2 in future? I am using geolite2 on server-side and there is some discrepancy caused by logstash. Specifying geolite2 db in geoip filter spits out errors.

dfaropennetwork commented 9 years ago

+1

markwalkom commented 9 years ago

+1

mshirley commented 9 years ago

+1

KrisBuytaert commented 9 years ago

+2

yinchuan commented 9 years ago

+1

xbgmsharp commented 9 years ago

👍

dforste commented 9 years ago

+1

IngaFeick commented 8 years ago

+1

Answer 1 · 2015-05-18T08:07:17.000Z

I developed a Logstash-filter-geoip2 to support maxmindDB/geoip2.
Actually the code was based on logstash-filter-geoip, I replaced its main code with official maxmindDB Java API.
It currently only support city information by now, limited by the ruby code I wrote in geoip2.rb
Note:This plugin doesn't include any maxmindDB database file.My Company has it but it will not be open to external.
I hope we can improve this plugin together.

Answer 2 · 2015-08-19T11:16:38.000Z

I was thinking of doing the same thing, could not see your code 404. Makes sense to build off the java api and support the various databases. Looking at geopip it should be a pretty trivial extension.

Answer 3 · 2015-08-19T20:53:33.000Z

Voting for GeoIP2 support in logstash-filter-geoip

Answer 4 · 2015-11-09T16:12:21.000Z

https://github.com/hp197/logstash-filter-geoip2

This is a initial release and WILL contain bugs.

Also note that the output of this plugin is slightly different.
I'm thinking about adding a option to have legacy output (which makes it backward compatible).

{
    "message" => "8.8.8.8",
    "@version" => "1",
    "@timestamp" => "2015-11-09T16:17:30.441Z",
    "type" => "test",
    "host" => "localhost",
    "geoip" => {
        "city" => "Mountain View",
        "continent" => {
            "code" => "NA",
            "name" => "North America"
        },
        "country" => {
            "iso_code" => "US",
            "name" => "United States"
        },
        "location" => {
            "latitude" => 37.386,
            "longitude" => -122.0838,
            "metro_code" => 807,
            "time_zone" => "America/Los_Angeles"
        },
        "postal" => "94040",
        "subdivisions" => [
            [0] {
                "iso_code" => "CA",
                "name" => "California"
            }
        ]
    }
}

Answer 5 · 2015-11-27T05:38:52.000Z

@hp197 👍

Answer 6 · 2016-03-03T02:42:30.000Z

Sorry for deleting my Logstash-filter-geoip2 project a year ago.Recently I recreated this project on Github, Your are free to use this logstash plugin.

This plugin has been running in my company's logstash instance more than one year with 2TB+ data every day.
GeoLite2-City.mmdb from official free maxmindDB database is included.
I will publish a gem to rubygems.org to make installation as easy as other logstash plugins in the future.

Need your feedback!

Answer 7 · 2016-03-03T02:46:18.000Z

@garyelephant thanks. Why not create a PR for this repo so we can update this plugin to GeoLite2? We can bump to major version then?

Its confusing to have 2 plugins for geoip

Answer 8 · 2016-03-03T08:01:50.000Z

@suyograo Make sense.I'm going to do this.

Answer 9 · 2016-03-03T18:25:02.000Z

Great new!

Why not this library https://github.com/yhirose/maxminddb? it's pure ruby i believe.

Thanks,

Answer 10 · 2016-03-03T18:32:44.000Z

@garyelephant woot! let us know if you need help. We added caching in #40.. so that and your java implementation would improve the performance here.

@ebuildy we could, but at this time, LS does not work well with anything other than JRuby, so why not use the java implementation?

Answer 11 · 2016-03-03T18:42:39.000Z

Ya agree, I don't know ;-)

I hope this new version will preload database like geoip can do (https://github.com/cjheath/geoip/blob/master/lib/geoip.rb#L239) I saw a big improvement with that.

Answer 12 · 2016-03-07T17:50:52.000Z

Hello any update on this? cheers

Answer 13 · 2016-03-07T18:28:36.000Z

@ebuildy nothing I'm aware of. I searched for geoip2 on open PRs in this repo and didn't find anything.

Answer 14 · 2016-03-07T21:10:20.000Z

Just tested geoip2, works fine as expected.

BTW, a good point => https://github.com/maxmind/GeoIP2-java#multi-threaded-use , this mean we don't need the Thread.current[@threadkey] trick, isn'it? (and share maxmind build-in cache between threads)

Answer 15 · 2016-03-08T02:20:55.000Z

@ebuildy Thanks for you advice, I will improve the codes.

Answer 16 · 2016-03-08T10:03:50.000Z

@garyelephant I did a small fork (https://github.com/ebuildy/logstash-filter-geoip2), moved to 2.4.0 because they added built-in cache.

BTW 2.6.0 is not compatible with logstash 2.2 because jackson library conflict (should be fixed with logstash 3.0).

Answer 17 · 2016-03-08T17:06:05.000Z

@ebuildy @garyelephant I am super excited to get this change into this plugin. Any ETA? :)

Answer 18 · 2016-03-08T17:28:26.000Z

You can give a try to my or garyelephant fork, any advice about how to integrate Java JAR vendor?

I am finishing logstash-plugins/logstash-filter-useragent#23 PR, then I could spend some time on it.

Answer 19 · 2016-03-09T02:15:39.000Z

@ebuildy Good, I'm going to make a pull request to logstash-filter-geoip this week.

Answer 20 · 2016-03-09T17:06:07.000Z

Great!
Le 9 mars 2016 03:15, "Gary Gao" notifications@github.com a écrit :

@ebuildy https://github.com/ebuildy Good, I'm going to make a pull
request to logstash-filter-geoip this week.

—
Reply to this email directly or view it on GitHub
#23 (comment)
.

Answer 21 · 2016-03-17T11:50:13.000Z

Hello, I have almost finish to implement all former fields:

https://github.com/ebuildy/logstash-filter-geoip2/blob/master/lib/logstash/filters/geoip.rb

Still a few issues when running spec tests, but I am on it!

Answer 22 · 2016-03-18T07:22:49.000Z

Thanks @ebuildy , I forked this repo and update its code to geoip2, Could you please commit your code in this repo, by pull request. After that, I can make a pull request to this repo to make geoip2 available to community.

One more thing, I move geoip2 jar dependencies to lib dir.

Answer 23 · 2016-03-18T15:32:55.000Z

Maybe we could copy field names from:

https://github.com/elastic/elasticsearch/blob/master/plugins/ingest-geoip/src/main/java/org/elasticsearch/ingest/geoip/GeoIpProcessor.java

?

Answer 24 · 2016-03-19T13:51:35.000Z

#64 I made a PR, Please take a look, @ebuildy ,@suyograo, Should I bump the version ?

Answer 25 · 2016-05-03T23:58:11.000Z

People also use this plugin with the legacy ASN database. The ASN number and names lookups are now provided by same geoip2 database. Can these please also be considered in the legacy naming support?

e.g.

"dest_asn": {
  "number": "AS16509",
  "asn": "Amazon.com, Inc."
}

Answer 26 · 2016-05-11T20:24:20.000Z

GeoIP2 and IPv6 support has been added in version 3.0.0 of this plugin for LS versions <= 2.x and 4.0.0 for LS version >= 5.x. Many thanks to @garyelephant and @ebuildy for helping out!