Query Regarding async-queue!

[vipinvkmenon]

This is a query, sorry if its the wrong place to ask.

In all documented examples (like the one below) we are batching as a best practice the async-queue!

(let [downstream (batch 1000 1
                        (async-queue! :agg {:queue-size     1e3
                                            :core-pool-size 4
                                            :max-pool-size  32}
                          (forward
                            (riemann.client/tcp-client
                              :host "127.0.0.1"))))]
  (streams
    ...
    ; Forward all events downstream to the aggregator.
    (where (service #"^riemann.*")
      downstream)))

So my understanding here is a set of 1000 events or 1 second worth of events (whichever first) is batched and sent downstream. For each batch sent down, a new executor service is created for every invocation of async-queue!. Is this right?
Then what is the significance of queue size here? There would always be only one job in the queue right?

For E.g: If batches were of 1000 and let's say the event rate is about 5k/s then, does that mean 5 threadpool-services (thereby TPE) are created?

[jarpy]

I believe that async-queue! is only invoked once for the entire lifetime of the config/core. It's evaluated in the let-binding at "compile time" and then you have it forever.

In our Riemann config, I wrote this comment:

  ;; We've now defined the queue. Importantly, we've done it only once, in the
  ;; let-binding. Now we can define a stream function that closes over the queue
  ;; so that we can send many events to the single queue instance.

Hope that helps (and that I understand it correctly).

[vipinvkmenon]

Thanks for the quick answer. Yes, you are absolutely right 😃 . Had a re-look at service! and it made all the more sense.