argoproj-labs/old-argo-dataflow

Source async commit/ack

Closed this issue · 1 comments

Currently, the architecture essentially assumes that messages are sunk synchronously. This is not what we want for several reasons:

  • Async Kafka producer.
  • Prevents buffering of sinking.

But note:

  • Some sources do not have ack, e.g. cron
  • Some sources must be acks sync, e.g. HTTP
  • Some sinks do not support async, e.g. HTTP
  • Some sinks only support async, e.g. log (yes, log writes are async and can be lost)
  • Some messages will not be sunk, e.g. due to filter op.

Stale issue message