logstash-plugins/logstash-filter-geoip

Add Support for additional GeoIP Databases

minars2 opened this issue · 1 comments

The ask in this feature request is to add support for more databases, specifically my case would be the DB-IP Location + ISP database, but it seems others would like the rest of the paid MaxMind databases as well.

It looks like the GeoIP filter can read the file but it expects a different schema for the data and therefore fails out when trying to use the DB-IP database in the GeoIP filter with the database option.

DB-IP's database schema is available here:
https://db-ip.com/db/format/ip-to-location-isp/mmdb.html

They also link to a MaxMind code library to read MMDB's which when I attempted to read the DB-IP database it wanted to use the Enterprise database reader, which at the moment doesn't appear to be supported by the GeoIP Filter.

I don't have access to a MaxMind Enterprise database so I couldn't check if to see the schema is similar or not, but the library seems to think so.

Also, it looks like the GeoIP filter does not recognize that MaxMind has added additional databases, such as the Enterprise, Anonymous IP, Connection Type, and Domain, all of which are paid MaxMind databases. Info for these databases can be found here: https://dev.maxmind.com/

There have been a few pull requests for adding support for some of these databases to the GeoIP Filter:
Anonymous IP database PR from 2018: #141
Domain database PR from 2020: https://github.com/logstash-plugins/logstash-filter-geoip/pull/162

Finally, here is a sample of data that I was able to pull using MaxMind's Python MMDB file reader inputting the IP 1.1.1.1 to the DB-IP database and Geolite2 City database:

https://github.com/maxmind/MaxMind-DB-Reader-python

Geolit2 Database:

{
    'continent': {
        'code': 'OC',
        'geoname_id': 6255151,
        'names': {
            'de': 'Ozeanien',
            'en': 'Oceania',
            'es': 'Oceanía',
            'fr': 'Océanie',
            'ja': 'オセアニア',
            'pt-BR': 'Oceania',
            'ru': 'Океания',
            'zh-CN': '大洋洲'
        }
    },
    'country': {
        'geoname_id': 2077456,
        'iso_code': 'AU',
        'names': {
            'de': 'Australien',
            'en': 'Australia',
            'es': 'Australia',
            'fr': 'Australie',
            'ja': 'オーストラリア',
            'pt-BR': 'Austrália',
            'ru': 'Австралия',
            'zh-CN': '澳大利亚'
        }
    },
    'location': {
        'accuracy_radius': 1000,
        'latitude': -33.494,
        'longitude': 143.2104,
        'time_zone': 'Australia/Sydney'
    },
    'registered_country': {
        'geoname_id': 2077456,
        'iso_code': 'AU',
        'names': {
            'de': 'Australien',
            'en': 'Australia',
            'es': 'Australia',
            'fr': 'Australie',
            'ja': 'オースト ラリア',
            'pt-BR': 'Austrália',
            'ru': 'Австралия',
            'zh-CN': '澳大利亚'
        }
    }
}

DB-IP Database:

{
    'city': {
        'geoname_id': 2147714,
        'names': {
            'de': 'Sydney',
            'en': 'Sydney',
            'es': 'Sídney',
            'fa': 'سیدنی',
            'fr': 'Sydney',
            'ja': 'シドニー',
            'ko': '시드니',
            'pt-BR': 'Sydney',
            'ru': 'Сидней',
            'zh-CN': '悉尼'
        }
    },
    'continent': {
        'code': 'OC',
        'geoname_id': 6255151,
        'names': {
            'de': 'Ozeanien',
            'en': 'Oceania',
            'es': 'Oceanía',
            'fa': 'اقیانوسیه',
            'fr': 'Océanie',
            'ja': 'オセアニア',
            'ko': '오세아니아',
            'pt-BR': 'Oceania',
            'ru': 'Океания',
            'zh-CN': '大洋洲'
        }
    },
    'country': {
        'geoname_id': 2077456,
        'is_in_european_union': False,
        'iso_code': 'AU',
        'names': {
            'de': 'Australien',
            'en': 'Australia',
            'es': 'Australia',
            'fa': 'استرالیا',
            'fr': 'Australie',
            'ja': 'オーストラリア',
            'ko': '오스트레일리아',
            'pt-BR': 'Austrália',
            'ru': 'Австралия',
            'zh-CN': '澳大利亚'
        }
    },
    'location': {
        'latitude': -33.8688,
        'longitude': 151.209,
        'time_zone': 'Australia/Sydney',
        'weather_code': 'ASXX0112'
    },
    'postal': {
        'code': '1001'
    },
    'subdivisions': [{
            'geoname_id': 2155400,
            'iso_code': 'NSW',
            'names': {
                'en': 'New South Wales',
                'fr': 'Nouvelle-Galles du Sud',
                'pt-BR': 'Nova Gales do Sul',
                'ru': 'Новый Южный Уэльс'
            }
        }
    ],
    'traits': {
        'autonomous_system_number': 13335,
        'autonomous_system_organization': 'Cloudflare, Inc.',
        'connection_type': 'Corporate',
        'isp': 'Cloudflare, Inc.'
    }
}

Thank you

pepl commented

Implemented with #208