Support at-least-once delivery semantics
Closed this issue · 2 comments
The IoTHub class is already abstracting the connection to the offset datastore. It's currently saving the offsets at periodic intervals in a separate process, but to support at-least-once semantics the offsets must be saved in stream after and only after the data has been processed.
In order to support this, it should be possible to expose a Sink for this purpose from the IoTHub object such that a developer can create a Graph that resembles the heuristic
hub.source ~> /** do work **/ ~> hub.offsetSink
This will require not only the specification of such a Sink, but also the ability for the hub object to load offsets without the periodic saving scheme.
I had a PR ready but the API has changed significantly enough I will have to rework a little.