logstash-plugins/logstash-output-google_bigquery

Batches should be published in paralel when using multiple pipeline workers

Closed this issue · 0 comments

Given that this plugin has concurrency :single (meaning a single instance will be shared by workers) the behavior of Batcher.enqueue causes the sending to be serialized across workers causing a bottleneck.

I believe the desired behavior is to have all workers sending data independently of each other whenever they have a batch ready.