Concept for 'event processors'
Closed this issue · 0 comments
stoewer commented
Although event processing via StreamAPI
works fine, it would still be nice to have some kind of high level wrapper around the StreamAPI
which provides the following features:
- Easy to use and intuitive API.
- Forwards all events to a provided callback with the signature
func(events []byte) error
of something similar. - The processor automatically commits cursors if the callback returns without error.
- Has the ability to spawn and manage multiple instances of stream API in order to be able to listen on multiple partitions in parallel.
- Optimizes event processing to get a higher throughput. For example process the next batch while the last cursor is being committed.
- It must be possible to cancel event processing gracefully.