logstash-plugins/logstash-filter-geoip

Private Subnets?

daniel10059 opened this issue · 5 comments

daniel10059 commented

Hi,

i have many private Subnets mapped to geolocations (lan, lot) in GeoIP Legacy Database Format.
Is it possible to use this database with this Plugin? I have tried to use version 3.0.0 of the plugin, but since i use logstash 5.6.0 i am unable to use the older version.

Is there any possibility to get a geoip2-database with private ips, or is there a way to use the old format?

cameronkerrnz commented

Have a look at https://blog.maxmind.com/2015/09/29/building-your-own-mmdb-database-for-fun-and-profit/, which I've been playing with internally for he last couple of days. It works (beware you need to set remove_reserved_networks => 0 when creating the $tree object otherwise it will silently ignore any private ranges you add.

Did I mention you'll need to write it in Perl -- you can make it easier on yourself by using your language of choice to emit data as JSON, then use Perl's JSON library to suck in the data and create the MMDB. As an example, I'm using Python to get data from an IP allocation spreadsheet, emitting that as JSON.

The biggest problem I have is that the geoip filter doesn't work with arbitrary databases (I have different fields, to model things such as which VLAN and Campus an IP belongs to). Looks like I'll have to use the same fields as the City database (or ASN database) --- or fork logstash-filter-geoip

sgreszcz commented

@cameronkerrnz

Thanks for the feedback on this. I spent hours today trying to geoIP enrich data using netflow data and source => "ipv4_src_addr" or source => "netflow.ipv4_src_addr" and I kept getting the _geoip_lookup_failure tag, despite the lookups seeming to work in python library with the same DB

import geoip2.database
reader = geoip2.database.Reader('/Users/me/Downloads/test.mmdb')
response = reader.city('10.50.172.174')

response.location.latitude
51.445324
response.location.longitude
-0.443962

response
geoip2.models.City({'city': {'names': {'en': ' FELTHAM'}}, 'continent': {'code': 'something'}, 'country': {'iso_code': ' UNITED KINGDOM', 'names': {'en': ' UNITED KINGDOM'}}, 'location': {'accuracy_radius': 1000, 'latitude': 51.445324, 'longitude': -0.443962}, 'traits': {'ip_address': '10.50.172.174'}}, ['en'])

I guess this is the problem - the structure of my DB (see above) not matching the City database or ASN database?

ViperGDC commented

I am having the same issues, is there a way we can see what the City or ASN structure looks like?

cameronkerrnz commented

Try using mmdblookup with the --verbose flag to see the structure.

https://maxmind.github.io/libmaxminddb/mmdblookup.html

I haven't tried to simulate the existing City or other schemas; I've only created my own, which I consume with my https://github.com/cameronkerrnz/logstash-filter-mmdb

ViperGDC commented

Thanks @cameronkerrnz! Wouldn't mmdbinspect be similar to using mmdblookup? I can get it to work fine using the Geolite-ASN.mmdb structure with the Logstash GeoIP filter but i guess its because it is a very simple structure. The problem lies when trying to mimic the city or country tree structure. I think it might have something to do with the names of the fields.

This is what is returned when i use mmdbinspect to get geo info for a specific ip...
image

using this structure i came up with the below code...

This is what my go code looks like (took a sample https://github.com/maxmind/mmdbwriter/blob/main/examples/asn-writer/main.go) to build the country.mmdb ...
image