dmachard/DNS-collector

Improve Clickhouse logger

Opened this issue · 0 comments

Improve the Clickhouse logger regarding flexibility and performance to at least also log DNS flags as we have it in the STDOUT logger:

Nov 13 07:17:12 resolver go-dnscollector[2595738]: {"network":{"family":"IPv4","protocol":"UDP","query-ip":"192.168.1.100","query-port":"51165","response-ip":"192.168.1.1","response-port":"53","ip-defragmented":false,"tcp-reassembled":false},"dns":{"length":54,"id":60280,"opcode":0,"rcode":"NOERROR","qname":"microsoft.com","qclass":"IN","qdcount":1,"ancount":0,"nscount":0,"arcount":1,"qtype":"TXT","flags":{"qr":false,"tc":false,"aa":false,"ra":false,"ad":true,"rd":true,"cd":false},"resource-records":{"an":[],"ns":[],"ar":[]},"malformed-packet":false},"edns":{"udp-size":1232,"rcode":0,"version":0,"dnssec-ok":0,"options":[{"code":10,"name":"COOKIE","data":"-"}]},"dnstap":{"operation":"CLIENT_QUERY","identity":"resolver","version":"dnsdist 1.9.6","timestamp-
rfc3339ns":"2024-11-13T06:17:10.76926877Z","latency":0,"extra":"-","policy-rule":"-","policy-type":"-","policy-match":"QNAME","policy-action":"NXDOMAIN","policy-value":"-","peer-name":"@","query-zone":"-"}}
Nov 13 07:17:12 resolver go-dnscollector[2595738]: {"network":{"family":"IPv4","protocol":"UDP","query-ip":"192.168.1.100","query-port":"51165","response-ip":"192.168.1.1","response-port":"53","ip-defragmented":false,"tcp-reassembled":false},"dns":{"length":70,"id":60280,"opcode":0,"rcode":"NOERROR","qname":"microsoft.com","qclass":"IN","qdcount":1,"ancount":0,"nscount":0,"arcount":1,"qtype":"TXT","flags":{"qr":true,"tc":true,"aa":false,"ra":true,"ad":false,"rd":true,"cd":false},"resource-records":{"an":[],"ns":[],"ar":[]},"malformed-packet":false},"edns":{"udp-size":1232,"rcode":0,"version":0,"dnssec-ok":0,"options":[{"code":10,"name":"COOKIE","data":"-"}]},"dnstap":{"operation":"CLIENT_RESPONSE","identity":"resolver","version":"dnsdist 1.9.6","timestamp-rfc3339ns":"2024-11-13T06:17:10.769545507Z","latency":0,"extra":"-","policy-rule":"-","policy-type":"-","policy-match":"QNAME","policy-action":"NXDOMAIN","policy-value":"-","peer-name":"@","query-zone":"-"}}

From the discussions:
#875 (comment)

The current ClickHouse logger is basic and not efficient under high load. To improve flexibility and performance, the logger needs the following updates:

  • Support for JSONEachRow insertion format

  • Batch insert functionality

  • A more structured data model