hazelcast/hazelcast-jet

Ability to set Output Vertex Queue size within pipeline API

ashley-taylor opened this issue · 0 comments

Hello,

We have a keyed Aggregation that computes an array of substantial objects.
The output of these aggregation goes straight into a flat map to explode this array.
It would be awesome if there were a way to return a Traverser from an aggregation.

Each object in the array can be in the 100's of MB, and the array can contain 100's of elements.

With the default queue settings, the ram requirements to queue this with the default settings are large.

We used the toDag method to convert the pipeline then manually set the outbound queue sizes.
This approach does works. It is a bit inconvenient and does rely on the deprecated toDag method.