ilanddev/syslogng_kafka

Memory Leak

groyee opened this issue · 4 comments

groyee commented

syslog-ng Kafka driver = latest.
Python version = 2.7.12
Operating System = Ubuntu 16.10

I am just sending simple messages to the syslog.

Huge memory leak. About 1MB every second

anguenot commented

Hey @groyee,

What version of syslog-ng are you using?

J.

groyee commented

I am currently using 3.7-3 but I tried it on 3.8.1-4 and it was the same.

The moment this plugin uses value-pairs in the destination as documented in the syslog-ng documentation there is a memory leak.

anguenot commented

Sorry for the late answer.

We using 3.7-3 as well on Ubuntu 16.04. We cannot see the leaks.

What do you mean by value-pairs in the destination? Can you give us an example please?

Thank you.

anguenot commented

Hey @groyee,

I reimplemented this module against confluent-kafka which uses the librdkafka C bindings.

See: https://github.com/confluentinc/confluent-kafka-python and https://github.com/edenhill/librdkafka

I did some benchmarks w/ syslog-ng 3.9.x and stress tests and cannot cause any leaks. If the Kafka broker is offline, the max memory it will use is 1M messages in the producer queue. After that, the messages will be discarded until the broker is back online.

I added a bunch of options as well: https://syslogng-kafka.readthedocs.io/en/latest/installation.html?

Closing this issue. Feel free to reach out if you playing with this again.