stoewer/go-nakadi

Concept for 'event processors'

Closed this issue · 0 comments

Although event processing via StreamAPI works fine, it would still be nice to have some kind of high level wrapper around the StreamAPI which provides the following features:

  • Easy to use and intuitive API.
  • Forwards all events to a provided callback with the signature func(events []byte) error of something similar.
  • The processor automatically commits cursors if the callback returns without error.
  • Has the ability to spawn and manage multiple instances of stream API in order to be able to listen on multiple partitions in parallel.
  • Optimizes event processing to get a higher throughput. For example process the next batch while the last cursor is being committed.
  • It must be possible to cancel event processing gracefully.