logstash-plugins/logstash-filter-geoip

GeoLite2-ASN Database doesn't work with 4.1.1 release.

robcowart opened this issue · 7 comments

robcowart commented

I am testing release 4.1.1 of the filter using both the GeoIP2-ISP.mmdb and GeoLite2-ASN.mmdb databases. The updated filter was added to Logstash using logstash-plugin update logstash-filter-geoip

The good news is that the GeoIP2-ISP database works great. This database was already supported in com.maxmind.geoip2 2.8.0 and com.maxmind.db 1.2.1, so it is expected that it should work once the functionality was added to the filter. The output from this database looks like this...

"geoip_asn_dst" => {
    "as_org" => "Vodafone Kabel Deutschland GmbH",
    "ip" => "123.123.123.123",
    "isp" => "Vodafone Kabel Deutschland",
    "organization" => "Vodafone Kabel Deutschland",
    "asn" => 31334
}

The GeoLite2-ASN database however does not work. I suspect this is due to the fact that support for this database was not included in the MaxMind libraries until com.maxmind.geoip2 2.9.0 and com.maxmind.db 1.2.2.

Notice that in lib/logstash-filter-geoip_jars.rb the old MaxMind libraries are still referenced, which may be the cause of the problem...

require_jar('com.maxmind.geoip2', 'geoip2', '2.8.0')
require_jar('com.maxmind.db', 'maxmind-db', '1.2.1')

This is where my Java and Logstash plugin development knowledge ends, so I haven't troubleshot the issue further. However since the plugin update installed only the old MaxMind libraries I strongly suspect this is the issue.

For completeness, here is the error seen when using the GeoLite2-ASN database.

[2017-05-20T12:16:30,878][ERROR][logstash.pipeline        ] Exception in pipelineworker, the pipeline stopped processing new events, please check your filter configuration and restart Logstash. {"exception"=>"Invalid attempt to open a GeoLite2-ASN database using the isp method", "backtrace"=>["com.maxmind.geoip2.DatabaseReader.get(com/maxmind/geoip2/DatabaseReader.java:146)", "com.maxmind.geoip2.DatabaseReader.isp(com/maxmind/geoip2/DatabaseReader.java:275)", "org.logstash.filters.GeoIPFilter.retrieveIspGeoData(org/logstash/filters/GeoIPFilter.java:317)", "org.logstash.filters.GeoIPFilter.handleEvent(org/logstash/filters/GeoIPFilter.java:130)", "java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:498)", "RUBY.filter(/Users/rob/local/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-geoip-4.1.1-java/lib/logstash/filters/geoip.rb:122)", "LogStash::Filters::Base.do_filter(/Users/rob/local/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:145)", "LogStash::Filters::Base.do_filter(/Users/rob/local/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:145)", "LogStash::Filters::Base.multi_filter(/Users/rob/local/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:164)", "LogStash::Filters::Base.multi_filter(/Users/rob/local/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:164)", "org.jruby.RubyArray.each(org/jruby/RubyArray.java:1613)", "LogStash::Filters::Base.multi_filter(/Users/rob/local/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:161)", "LogStash::Filters::Base.multi_filter(/Users/rob/local/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:161)", "LogStash::FilterDelegator.multi_filter(/Users/rob/local/usr/share/logstash/logstash-core/lib/logstash/filter_delegator.rb:43)", "LogStash::FilterDelegator.multi_filter(/Users/rob/local/usr/share/logstash/logstash-core/lib/logstash/filter_delegator.rb:43)", "RUBY.initialize((eval):499931)", "org.jruby.RubyArray.each(org/jruby/RubyArray.java:1613)", "RUBY.initialize((eval):499926)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:281)", "RUBY.initialize((eval):499947)", "org.jruby.RubyArray.each(org/jruby/RubyArray.java:1613)", "RUBY.initialize((eval):499942)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:281)", "RUBY.initialize((eval):500009)", "org.jruby.RubyArray.each(org/jruby/RubyArray.java:1613)", "RUBY.initialize((eval):500005)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:281)", "RUBY.initialize((eval):500685)", "org.jruby.RubyArray.each(org/jruby/RubyArray.java:1613)", "RUBY.initialize((eval):500680)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:281)", "RUBY.initialize((eval):500720)", "org.jruby.RubyArray.each(org/jruby/RubyArray.java:1613)", "RUBY.initialize((eval):500706)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:281)", "RUBY.filter_func((eval):22418)", "LogStash::Pipeline.filter_batch(/Users/rob/local/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:370)", "LogStash::Pipeline.filter_batch(/Users/rob/local/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:370)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:281)", "LogStash::Util::WrappedSynchronousQueue::ReadBatch.each(/Users/rob/local/usr/share/logstash/logstash-core/lib/logstash/util/wrapped_synchronous_queue.rb:224)", "LogStash::Util::WrappedSynchronousQueue::ReadBatch.each(/Users/rob/local/usr/share/logstash/logstash-core/lib/logstash/util/wrapped_synchronous_queue.rb:224)", "org.jruby.RubyHash.each(org/jruby/RubyHash.java:1342)", "LogStash::Util::WrappedSynchronousQueue::ReadBatch.each(/Users/rob/local/usr/share/logstash/logstash-core/lib/logstash/util/wrapped_synchronous_queue.rb:223)", "LogStash::Util::WrappedSynchronousQueue::ReadBatch.each(/Users/rob/local/usr/share/logstash/logstash-core/lib/logstash/util/wrapped_synchronous_queue.rb:223)", "LogStash::Pipeline.filter_batch(/Users/rob/local/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:369)", "LogStash::Pipeline.filter_batch(/Users/rob/local/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:369)", "RUBY.worker_loop(/Users/rob/local/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:350)", "RUBY.start_workers(/Users/rob/local/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:317)", "java.lang.Thread.run(java/lang/Thread.java:745)"]}
Exception in thread "[main]>worker7" java.lang.UnsupportedOperationException: Invalid attempt to open a GeoLite2-ASN database using the isp method
	at com.maxmind.geoip2.DatabaseReader.get(com/maxmind/geoip2/DatabaseReader.java:146)
	at com.maxmind.geoip2.DatabaseReader.isp(com/maxmind/geoip2/DatabaseReader.java:275)
	at org.logstash.filters.GeoIPFilter.retrieveIspGeoData(org/logstash/filters/GeoIPFilter.java:317)
	at org.logstash.filters.GeoIPFilter.handleEvent(org/logstash/filters/GeoIPFilter.java:130)
	at java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:498)
	at RUBY.filter(/Users/rob/local/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-geoip-4.1.1-java/lib/logstash/filters/geoip.rb:122)
	at LogStash::Filters::Base.do_filter(/Users/rob/local/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:145)
	at LogStash::Filters::Base.do_filter(/Users/rob/local/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:145)
	at LogStash::Filters::Base.multi_filter(/Users/rob/local/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:164)
	at LogStash::Filters::Base.multi_filter(/Users/rob/local/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:164)
	at org.jruby.RubyArray.each(org/jruby/RubyArray.java:1613)
	at LogStash::Filters::Base.multi_filter(/Users/rob/local/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:161)
	at LogStash::Filters::Base.multi_filter(/Users/rob/local/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:161)
	at LogStash::FilterDelegator.multi_filter(/Users/rob/local/usr/share/logstash/logstash-core/lib/logstash/filter_delegator.rb:43)
	at LogStash::FilterDelegator.multi_filter(/Users/rob/local/usr/share/logstash/logstash-core/lib/logstash/filter_delegator.rb:43)
	at RUBY.initialize((eval):499931)
	at org.jruby.RubyArray.each(org/jruby/RubyArray.java:1613)
	at RUBY.initialize((eval):499926)
	at org.jruby.RubyProc.call(org/jruby/RubyProc.java:281)
	at RUBY.initialize((eval):499947)
	at org.jruby.RubyArray.each(org/jruby/RubyArray.java:1613)
	at RUBY.initialize((eval):499942)
	at org.jruby.RubyProc.call(org/jruby/RubyProc.java:281)
	at RUBY.initialize((eval):500009)
	at org.jruby.RubyArray.each(org/jruby/RubyArray.java:1613)
	at RUBY.initialize((eval):500005)
	at org.jruby.RubyProc.call(org/jruby/RubyProc.java:281)
	at RUBY.initialize((eval):500685)
	at org.jruby.RubyArray.each(org/jruby/RubyArray.java:1613)
	at RUBY.initialize((eval):500680)
	at org.jruby.RubyProc.call(org/jruby/RubyProc.java:281)
	at RUBY.initialize((eval):500720)
	at org.jruby.RubyArray.each(org/jruby/RubyArray.java:1613)
	at RUBY.initialize((eval):500706)
	at org.jruby.RubyProc.call(org/jruby/RubyProc.java:281)
	at RUBY.filter_func((eval):22418)
	at LogStash::Pipeline.filter_batch(/Users/rob/local/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:370)
	at LogStash::Pipeline.filter_batch(/Users/rob/local/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:370)
	at org.jruby.RubyProc.call(org/jruby/RubyProc.java:281)
	at LogStash::Util::WrappedSynchronousQueue::ReadBatch.each(/Users/rob/local/usr/share/logstash/logstash-core/lib/logstash/util/wrapped_synchronous_queue.rb:224)
	at LogStash::Util::WrappedSynchronousQueue::ReadBatch.each(/Users/rob/local/usr/share/logstash/logstash-core/lib/logstash/util/wrapped_synchronous_queue.rb:224)
	at org.jruby.RubyHash.each(org/jruby/RubyHash.java:1342)
	at LogStash::Util::WrappedSynchronousQueue::ReadBatch.each(/Users/rob/local/usr/share/logstash/logstash-core/lib/logstash/util/wrapped_synchronous_queue.rb:223)
	at LogStash::Util::WrappedSynchronousQueue::ReadBatch.each(/Users/rob/local/usr/share/logstash/logstash-core/lib/logstash/util/wrapped_synchronous_queue.rb:223)
	at LogStash::Pipeline.filter_batch(/Users/rob/local/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:369)
	at LogStash::Pipeline.filter_batch(/Users/rob/local/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:369)
	at RUBY.worker_loop(/Users/rob/local/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:350)
	at RUBY.start_workers(/Users/rob/local/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:317)
	at java.lang.Thread.run(java/lang/Thread.java:745)
pmontrasio commented

I just run into this while working with logstash.

There seems to be something wrong with the code in src/main/java/org/logstash/filters/GeoIPFilter.java

I quote the relevant parts from all over the file

  ...
  private static final String ASN_LITE_DB_TYPE = "GeoLite2-ASN";
  private static final String ISP_DB_TYPE = "GeoIP2-ISP";
  ...
      switch (databaseReader.getMetadata().getDatabaseType()) {
        ...
        case ASN_LITE_DB_TYPE:
        case ISP_DB_TYPE:
          geoData = retrieveIspGeoData(ipAddress);
        ...
  private Map<String, Object> retrieveIspGeoData(InetAddress ipAddress) throws GeoIp2Exception, IOException {
    IspResponse response = databaseReader.isp(ipAddress);

The last line is the one that breaks.
If there is no support for the GeoLite2-ASN database maybe it's better to display an error message in the switch for case ASN_LITE_DB_TYPE. "The database GeoLite2-ASN is not supported" would be more clear than the current error.

However at https://github.com/maxmind/GeoIP2-java/blob/master/src/main/java/com/maxmind/geoip2/DatabaseReader.java I see

    public AsnResponse asn(InetAddress ipAddress) throws IOException,
            GeoIp2Exception {
        return this.get(ipAddress, AsnResponse.class, "GeoLite2-ASN");
    }

so there is already support for the ASN database. They added it with this commit maxmind/GeoIP2-java@ddf66e0 on April 19. That's version 2.9.0 of GeoIP2. This project is using 2.8.0.

  • 👍2
jordansissel commented

@suyograo can you look into this?

  • 👍1
robcowart commented

Is there any timeline for looking at this?

t3chn0m4g3 commented

Yes please :) Any progress is very much appreciated :)

suyograo commented

There's a PR here: #120. I'm hoping to merge it in the next couple of days.

  • 🎉2
t3chn0m4g3 commented

@suyograo This is awesome! Thank you so much!

pmontrasio commented

Thanks!

I pulled master and rebuilt. There are some quirks left:

  1. bundle exec rake vendor doesn't download the ASN database. It should be fixed by
diff --git a/vendor.json b/vendor.json
index 13b6439..f489e80 100644
--- a/vendor.json
+++ b/vendor.json
@@ -2,5 +2,10 @@
     {
         "url": "http://geolite.maxmind.com/download/geoip/database/GeoLite2-City.mmdb.gz",
         "sha1": "dc594fb45880bb33767f3280f2f108389f24db22"
+    },
+    {
+        "url": "http://geolite.maxmind.com/download/geoip/database/GeoLite2-ASN.tar.gz",
+        "files": "GeoLite2-ASN.mmdb",
+        "sha1": "ce70f043bf71602a211cef3565560e17d3543609"
     }
 ]

(I verified the md5 and then computed the sha1)

but...

  1. It can't extract the ASN database anyway because of this issue I opened right now to Elastic elastic/logstash-devutils#71

  2. I hacked a fix for that (see the issue) but bundle exec rspec fails with

$ bundle exec rspec
--- jar coordinate com.fasterxml.jackson.core:jackson-databind already loaded with version 2.7.4 - omit version 2.7.3
--- jar coordinate com.fasterxml.jackson.core:jackson-annotations already loaded with version 2.7.0 - omit version 2.7.3
--- jar coordinate com.fasterxml.jackson.module:jackson-module-afterburner already loaded with version 2.7.4 - omit version 2.7.3
--- jar coordinate com.fasterxml.jackson.core:jackson-core already loaded with version 2.7.4 - omit version 2.7.3
/home/me/.rvm/gems/jruby-9.1.8.0@logstash-filter-geoip/gems/logstash-core-5.4.0-java/lib/logstash/api/commands/stats.rb:57: warning: key (SymbolNode:heap_used_in_bytes 60) is duplicated and overwritten on line 57
/home/me/.rvm/gems/jruby-9.1.8.0@logstash-filter-geoip/gems/logstash-core-5.4.0-java/lib/logstash/api/commands/node.rb:41: warning: key (SymbolNode:vm_name 45) is duplicated and overwritten on line 43
Sending Logstash's logs to  which is now configured via log4j2.properties
Run options: exclude {:redis=>true, :socket=>true, :performance=>true, :couchdb=>true, :elasticsearch=>true, :elasticsearch_secure=>true, :export_cypher=>true, :integration=>true, :windows=>true}

Randomized with seed 53120
F.....................

Failures:

  1) LogStash::Filters::GeoIP GeoIP2-ASN database "{"ip":"8.8.8.8"}" when processed
     Got 0 failures and 2 other errors:

     1.1) Failure/Error: expect(subject.get("geoip")).not_to be_empty
          
          LogStash::ConfigurationError:
            Something is wrong with your configuration.
          # /home/me/.rvm/gems/jruby-9.1.8.0@logstash-filter-geoip/gems/logstash-core-5.4.0-java/lib/logstash/config/mixin.rb:130:in `config_init'
          # /home/me/.rvm/gems/jruby-9.1.8.0@logstash-filter-geoip/gems/logstash-core-5.4.0-java/lib/logstash/filters/base.rb:128:in `initialize'
          # /home/me/.rvm/gems/jruby-9.1.8.0@logstash-filter-geoip/gems/logstash-core-5.4.0-java/lib/logstash/filter_delegator.rb:21:in `initialize'
          # /home/me/.rvm/gems/jruby-9.1.8.0@logstash-filter-geoip/gems/logstash-core-5.4.0-java/lib/logstash/pipeline.rb:96:in `plugin'
          # (eval):8:in `<eval>'
          # /home/me/.rvm/gems/jruby-9.1.8.0@logstash-filter-geoip/gems/logstash-core-5.4.0-java/lib/logstash/pipeline.rb:63:in `initialize'
          # /home/me/.rvm/gems/jruby-9.1.8.0@logstash-filter-geoip/gems/logstash-core-5.4.0-java/lib/logstash/pipeline.rb:145:in `initialize'
          # /home/me/.rvm/gems/jruby-9.1.8.0@logstash-filter-geoip/gems/logstash-devutils-1.3.3-java/lib/logstash/devutils/rspec/logstash_helpers.rb:134:in `new_pipeline_from_string'
          # /home/me/.rvm/gems/jruby-9.1.8.0@logstash-filter-geoip/gems/logstash-devutils-1.3.3-java/lib/logstash/devutils/rspec/logstash_helpers.rb:39:in `block in pipeline'
          # /home/me/.rvm/gems/jruby-9.1.8.0@logstash-filter-geoip/gems/logstash-devutils-1.3.3-java/lib/logstash/devutils/rspec/logstash_helpers.rb:50:in `block in results'
          # /home/me/.rvm/gems/jruby-9.1.8.0@logstash-filter-geoip/gems/logstash-devutils-1.3.3-java/lib/logstash/devutils/rspec/logstash_helpers.rb:68:in `block in subject'
          # ./spec/filters/geoip_spec.rb:282:in `block in (root)'
          # /home/me/.rvm/gems/jruby-9.1.8.0@logstash-filter-geoip/gems/rspec-wait-0.0.9/lib/rspec/wait.rb:46:in `block in (root)'

     1.2) Failure/Error: raise LogStash::ConfigurationError,
          
          LogStash::ConfigurationError:
            Something is wrong with your configuration.
          # /home/me/.rvm/gems/jruby-9.1.8.0@logstash-filter-geoip/gems/logstash-core-5.4.0-java/lib/logstash/config/mixin.rb:130:in `config_init'
          # /home/me/.rvm/gems/jruby-9.1.8.0@logstash-filter-geoip/gems/logstash-core-5.4.0-java/lib/logstash/filters/base.rb:128:in `initialize'
          # /home/me/.rvm/gems/jruby-9.1.8.0@logstash-filter-geoip/gems/logstash-core-5.4.0-java/lib/logstash/filter_delegator.rb:21:in `initialize'
          # /home/me/.rvm/gems/jruby-9.1.8.0@logstash-filter-geoip/gems/logstash-core-5.4.0-java/lib/logstash/pipeline.rb:96:in `plugin'
          # (eval):8:in `<eval>'
          # /home/me/.rvm/gems/jruby-9.1.8.0@logstash-filter-geoip/gems/logstash-core-5.4.0-java/lib/logstash/pipeline.rb:63:in `initialize'
          # /home/me/.rvm/gems/jruby-9.1.8.0@logstash-filter-geoip/gems/logstash-core-5.4.0-java/lib/logstash/pipeline.rb:145:in `initialize'
          # /home/me/.rvm/gems/jruby-9.1.8.0@logstash-filter-geoip/gems/logstash-devutils-1.3.3-java/lib/logstash/devutils/rspec/logstash_helpers.rb:134:in `new_pipeline_from_string'
          # /home/me/.rvm/gems/jruby-9.1.8.0@logstash-filter-geoip/gems/logstash-devutils-1.3.3-java/lib/logstash/devutils/rspec/logstash_helpers.rb:39:in `block in pipeline'
          # /home/me/.rvm/gems/jruby-9.1.8.0@logstash-filter-geoip/gems/logstash-devutils-1.3.3-java/lib/logstash/devutils/rspec/logstash_helpers.rb:65:in `block in sample'
          # /home/me/.rvm/gems/jruby-9.1.8.0@logstash-filter-geoip/gems/rspec-wait-0.0.9/lib/rspec/wait.rb:46:in `block in (root)'

Finished in 1.47 seconds (files took 5.5 seconds to load)
22 examples, 1 failure

Failed examples:

rspec ./spec/filters/geoip_spec.rb:281 # LogStash::Filters::GeoIP GeoIP2-ASN database "{"ip":"8.8.8.8"}" when processed

Randomized with seed 53120