fluent-plugins-nursery/geoip2_c

memory leak issue when bunch of IPs lookup

y-ken opened this issue · 7 comments

y-ken commented

@okkez Could you please take a look?
I have got question from fluent-plugin-geoip user. And also I have requested to share the configuration what they use.
It seems much variation of IP process. Does it stores cache in somewhere?
How about introduce unset memory or LRU or limit cache count?

Our service connects on various ip
I suspect that the geoip plugin is caching memory for every ip.
The fluentd plugin process is steadily increasing memory usage by about 400 megabytes per week, and recently it has exceeded 10 gigabytes.

okkez commented

This library does not use any cache mechanism explicitly.
Maybe it has some memory leak issues.

andropler commented

This is a memory leak in our application td-agent.conf file.

<filter myapp.*>
    @type geoip
    geoip_lookup_keys ip
    <record>
        geoip '{"location": { "lon":${location.longitude["ip"]}, "lat":${location.latitude["ip"]}}, "country":${country.iso_code["ip"]}, "countryName":${country.names.en["ip"]}, "postalCode": ${postal.code["ip"]}, "city":${city.names.en["ip"]}}'
    </record>
    skip_adding_null_record true
</filter>

OS is centos 7.5 and the plugin installation was done using the following command
/usr/sbin/td-agent-gem install fluent-plugin-geoip

okkez commented

Cached objects here using Hash object...

y-ken commented

Oh, indeed. Do we have a chance to improve memory consumption?

okkez commented

I've confirmed memory consumption with various IP addresses while running Fluentd with fluent-plugin-geoip (geoip2_c backend).
I'm investigating reproducible code without Fluentd.

  • 👍2
okkez commented

Released v0.3.3.

  • 👍2
y-ken commented

Thank you! I have announced for the user how to apply latest geoip2_c with fluent-plugin-geoip.
I have just told to install latest geoip2_c. sudo td-agent-gem install geoip2_c

  • 👍1