MaterializeInc/datagen

Feature: Improve Producer Performance

Closed this issue · 2 comments

Is your feature request related to a problem? Please describe.

Right now, we create and destroy a producer on each iteration. It would be a little more efficient to create a single producer and then disconnect when the program exits.

We could also consider using node-rdkafka (maintained by Blizzard) which based on the more performant librdkafka library

Maintaining the producer across iterations is something we should do regardless of the library.

Whether or not we switch to node-rdkafka is less important to me. This is not so high throughput in the medium term that this sort of performance is important. We should value ease of use and functionality over all else.