riemann/riemann

`influxdb2` keeps accumulating background threads

Closed this issue · 3 comments

node13h commented

Describe the bug
Posting to InfluxDB using influxdb2 keeps accumulating background threads until Riemann or machine crashes.

To Reproduce
Steps to reproduce the behavior:
1.Configure Riemann to send data from few dozen reasonably active streams using influxdb2
2. Wait and observe accumulating threads using VisualVM.

Expected behavior
Stable operation.

Screenshots
image
image

Background (please complete the following information):

  • OS: CentOS 7
  • Java/JVM version 1.8.0_322-b06
  • Riemann version 0.3.8

Additional context
WriteApi creates a background thread and is supposed to run as a singleton. riemann.influxdb2.post-data creates a new instance of WriteApi on each invocation.
Using WriteApiBlocking fixes the issue.

jarpy commented

It's not super-obvious, but the docs for influxdb2 suggest that:

For performance, you should wrap this stream with batch or an asynchronous queue.

You can use the async queue to call influxdb2 only once. Below is a sketch based on how I do this for Logstash. Note that I have not tested this sketch (sorry), but I think the same principles will apply for your use-case.

;; Establish a dedicated queue and thread(s) for sending metrics to InfluxDB.
(let [influx-queue (async-queue!
                    :influx-output-queue {:queue-size 1000 :core-pool-size 1 :max-pool-size 1}
                    (influxdb2 {:host "influxdb.example.com"
                                :organization "riemann"
                                :bucket "riemann"
                                :token "riemann"}))]

  ;; We've now defined the queue. Importantly, we've done it only once, in the
  ;; let-binding. Now we can define a stream function that closes over the queue
  ;; so that we can send many events to the single queue instance.
  (defn send-to-influx
    "A stream for sending events to influxdb2 via an async queue."
    [& children]
    (fn [event]
      (call-rescue event (conj children influx-queue)))))
node13h commented

@jarpy thanks, but I've already got it wrapped in async-queue! once. The VisualVM screenshots above were taken while influxdb2 was wrapped in async-queue!.
The problem is not in influxdb2, but in post-data which keeps creating a new instances of WriteApi on each invocation.

My config:

influxdb-batch-sender (batch 100 1
    (async-queue! :agg {:queue-size 1000
                        :core-pool-size 2
                        :max-pool-size 16}
                  (influxdb2 influxdb-creds)))

EDIT: formatting, and clarifications.

jarpy commented

Ah! Thanks for explaining. I was making assumptions based on my own experience, not yours. Sorry for the trouble.

  • 👍2