dmachard/DNS-collector

High Memory Usage with Clickhouse logger

Opened this issue · 3 comments

Resha777 commented

Hello @dmachard ,

I'm currently running a dns-collector container with the below config.yml

global:
  worker:
    interval-monitor: 10
    buffer-size: 2000
  trace:
    verbose: true

pipelines:
  - name: dnsdist
    dnstap:
      listen-ip: 127.0.0.1
      listen-port: 6000
      chan-buffer-size: 65535
    routing-policy:
      forward: [ clk ]
      dropped: []

  - name: clk
    clickhouse:
      url: "http://XXXX:8123"
      user: "dns_edge"
      password: "XXXX"
      table: "alldns_buf"
      database: "dns"
      chan-buffer-size: 65535

I configured my dnsdist to send some domains using dnstap , all is working great but the memory consumption of the container is not normal , the container has been running for a 1 day and the memory is around 3.5GB and it's still going up eating the vm's memory.

Is there an option that we can put to limit the memory usage ?

Appreciate you support.

dmachard commented

Hello,

Could you please share the output logs? Additionally, could you let us know the volume of DNS traffic you're handling?

Also, could you try removing the chan-buffer-size setting from your configuration and use the default value of buffer-size: 8192 in the global section?

As mentioned in this issue, improvements are still needed for the ClickHouse integration.

Resha777 commented

Hello,

here are the logs from the container

podman logs dnscollector 
INFO: 2024/12/25 09:28:00.079927 main - version=v1.3.0 revision=
INFO: 2024/12/25 09:28:00.080019 main - running in pipeline mode
INFO: 2024/12/25 09:28:00.080620 worker - [dnsdist] dnstap - enabled
INFO: 2024/12/25 09:28:00.081911 worker - [dnsdist] dnstap - starting monitoring - refresh every 10s
INFO: 2024/12/25 09:28:00.082937 worker - [clk] clickhouse - enabled
INFO: 2024/12/25 09:28:00.083296 main - routing (policy=forward) stanza=[dnsdist] to stanza=[clk]
INFO: 2024/12/25 09:28:00.083370 worker - [clk] clickhouse - starting monitoring - refresh every 10s
INFO: 2024/12/25 09:28:00.083760 worker - [dnsdist] dnstap - starting data collection
INFO: 2024/12/25 09:28:00.083935 worker - [clk] clickhouse - starting data collection
INFO: 2024/12/25 09:28:00.084346 worker - [clk] clickhouse - logging has started
INFO: 2024/12/25 09:28:00.084484 worker - [dnsdist] dnstap - listening on 127.0.0.1:6000
INFO: 2024/12/25 09:28:01.991703 worker - [dnsdist] dnstap - conn #1 - new connection from 127.0.0.1:58696 (localhost)
INFO: 2024/12/25 09:28:01.991716 worker - [dnsdist] (conn #1) dnstap processor - enabled
INFO: 2024/12/25 09:28:01.991925 worker - [dnsdist] (conn #1) dnstap processor - starting monitoring - refresh every 10s
INFO: 2024/12/25 09:28:01.991974 worker - [dnsdist] (conn #1) dnstap processor - starting data collection
INFO: 2024/12/25 09:28:01.992059 worker - [dnsdist] dnstap - conn #1 - receiver framestream initialized

Also regarding the DNS traffic , we are currently testing with few domains in dnsdist, about 200 hits per hour so it's basically nothing to mention

I will try to remove the chan-buffer-size and monitor further.

mahdiadnan commented

Hi @dmachard would valgrind profiler be helpful to debug this memory leak?

  • 👍1