Source async commit/ack
Closed this issue · 1 comments
alexec commented
Currently, the architecture essentially assumes that messages are sunk synchronously. This is not what we want for several reasons:
- Async Kafka producer.
- Prevents buffering of sinking.
But note:
- Some sources do not have ack, e.g. cron
- Some sources must be acks sync, e.g. HTTP
- Some sinks do not support async, e.g. HTTP
- Some sinks only support async, e.g. log (yes, log writes are async and can be lost)
- Some messages will not be sunk, e.g. due to filter op.
github-actions commented
Stale issue message