RegioHelden/django-kafka

Implement inter-topic dependency management

Opened this issue · 0 comments

Add an optional functionality for inter-topic dependency management, to resolve the following issues:

  1. If an Event 1 for entity A fails in a topic and is sent to the non-blocking retry or dead letter queue, it may be desired that Event 2 for the same entity A should also not be processed until Event 1 succeeds (Event 2 should be put in a holding queue). Essentially, the order of events for the same entity needs to be respected.
  2. It could be that messages in some topics should not be processed before messages in other topics. These dependent messages should be put in a holding queue until their dependency in the other topic has been processed.

"Entity" can be determined by the message key normally, but ideally it should be overideable.

Sources for implementation:

  1. https://www.confluent.io/blog/error-handling-patterns-in-kafka/#pattern-4
  2. https://medium.com/altitudehq/kafka-retries-and-maintain-order-of-retry-events-313482044351 - see "Maintain order of retry events"
  3. https://www.youtube.com/watch?v=FO2ptQNQKhM&ab_channel=DanielTammadge