chr0mag/geoipsets

[MaxMind] Improve performance of set generation

chr0mag opened this issue · 0 comments

The MaxMind set generation logic is currently very simple. It just loops through the list of country IP blocks for the respective address families (ie. GeoLite2-Country-Blocks-IPv4.csv and GeoLite2-Country-Blocks-IPv6.csv) and writes to each CC.ipvX file as it goes.

This results in repeated appending to different files, causing lots of (slow) disk IO. A more performant solution would be to build a Map with the country codes as keys and the value a list of IP subnets belonging to that country. This would generate the country sets in memory (fast) rather than building them incrementally on disk. We can then write out the complete set files in one go at the end of processing, significantly reducing disk IO. This is what the DbIP provider does.