Possible memory leak

Question

Possible memory leak

Closed this issue 7 years ago · 11 comments

Hi,

I launch fw1-loggrabber in a docker container to pull firewall flow logs.
The container crashs after approximatively 24 hours because the process fw1-loggrabber consumed all the memory available and the kernel kill it (OOM killer).

The virtual machine used to launch the docker container has 2 GB of RAM. I see the fw1-loggrabber process memory consumption grows linearly.

I also launch another docker container (in the same docker host) to pull firewall audit logs. This container doesn't crash. It consumes only 2 MB of RAM.

Should i have more memory available in my virtual machine ? If yes, how much memory ?
If no, it seems to be a memory leak :(.

Can you help me please ?
Thanks.
Gabriel MACHADO.

Answer 1 · 2017-09-07T10:10:07.000Z

Could you please share the content of your fw1-loggrabber.conf configuration file?

Answer 2 · 2017-09-07T12:17:23.000Z

You can see below configuration files for each containers:

cat fw1-loggrabber_flow.conf

DEBUG_LEVEL=0
FW1_OUTPUT=logs
FW1_TYPE=ng
FW1_MODE=normal
ONLINE_MODE=yes
RESOLVE_MODE=no
LOGGING_CONFIGURATION=file
OUTPUT_FILE_PREFIX=fw1-loggrabber_flow
OUTPUT_FILE_ROTATESIZE=1073741824

cat fw1-loggrabber_audit.conf

DEBUG_LEVEL=0
FW1_OUTPUT=logs
FW1_TYPE=ng
FW1_MODE=audit
ONLINE_MODE=yes
RESOLVE_MODE=no
LOGGING_CONFIGURATION=file
OUTPUT_FILE_PREFIX=fw1-loggrabber_audit
OUTPUT_FILE_ROTATESIZE=1073741824

Answer 3 · 2017-09-07T12:47:25.000Z

We've had mixed feedback about the internal file rotation feature. You have currently set ~1GB as the maximum size of the output file before it is rotated.

Could you please try disabling the rotation as shown here? You must then use logrotate or equivalent to handle the rotation.

https://github.com/certego/fw1-loggrabber/blob/master/fw1-loggrabber.conf#L40-L42

Answer 4 · 2017-09-07T14:07:00.000Z

Thank you for your help.

I disabled size rotation file. The rotation is now handled by logrotate. It rotates the file when it reaches a size greater than 10 MB. In fact, the file is rotated when the file size is between 12 MB and 17 MB.

Unfortunately the memory usage is still growing.

Answer 5 · 2017-09-07T14:31:52.000Z

Thanks. A question before we do further checks: are you using latest commit ab8e506 from master or are you using release v2.1?

Answer 6 · 2017-09-07T14:38:07.000Z

I use the latest commit ab8e506

Answer 7 · 2017-09-07T15:50:41.000Z

Could you please try if the above fix works for you?

Answer 8 · 2017-09-08T08:56:14.000Z

Good work.

The process consumes now less than 3 MB of memory.
I will reenable size rotation file. I 'll close the issue if there is no more problem.

Thank you very much.

Answer 9 · 2017-09-08T08:59:11.000Z

Ok thanks for reporting!

Answer 10 · 2017-09-08T12:45:03.000Z

The issue about memory leak can be closed. It also works with size rotation file.

I have just one question.
What happened if i stop the process during a lapse of time ?
When i will restart it, will i receive the missing logs or just the logs produced at the moment ?

Answer 11 · 2017-09-08T14:39:23.000Z

Thanks, I am closing the issue.

Please note this is not Check Point customer support so my answer might be inaccurate/incorrect. To my understanding, if you stop pulling logs they are simply kept in a queue; when you restart pulling them again, you should be able to read them all from the queue (cfr. ONLINE_MODE).