Support unfixed backoff strategy

[ta7uw]

In the retrying mechanism of decaton processor, currently, the fixed backoff is only supported.
It is preferable to increase the interval as the number of retries increases rather than the fixed interval.

I guess it is useful to support exponential backoff, fibonacci backoff, and randomized backoff.

[ocadaruma]

Yeah, we know some users want this feature (exponential backoff) but it could be challenging because:

• In Decaton (Kafka), mixing tasks with different backoff in single partition doesn't make much sense.
  • Let's say we queue retry task with backoff-500ms to retry-topic-0
  • Next, we queue retry task with backoff-100ms to retry-topic-0
  • backoff-500ms will be consumed first. Decaton sleeps the processor-thread for 500ms till start processing
  • So, backoff-100ms will be started after processing backoff-500ms, while the backoff is 100ms.

This is inevitable because Kafka's partition behaves as queue.

Possible ideas to solve this:

• Having another internal queueing mechanism inside Decaton for different backoffs
  • However we can hold only limited tasks due to memory bound. So it doesn't scale for high-traffic or long-backoff cases
• Having different topics for backoffs
  • like retry-topic-backoff100ms, retry-topic-back-off500ms,...
  • I can find some cases on the web that achieving flexible backoff using this strategy
  • However, we need to prepare many topics

[ta7uw]

Thanks for your explanation.
It makes sense to me that supporting unfixed backoff would be challenging.