faust-streaming/faust

Is there a way to assign the number of partitions on table dynamically relying on topic's number of partitions

dolshevsk opened this issue · 1 comments

Checklist

  • I have included information about relevant versions
  • I have verified that the issue persists when using the master branch of Faust.

Steps to reproduce

I want to know if there is a way to assign the number of partitions on table dynamically relying on topic's number of partitions. In my app I consume from external topics which doesn't belong to me which has different number of partitions in different environment and their number can be changed in future, but I do modify table under those topics and thus the number of partitions should be equal.

Expected behavior

I wanted to achieve it with something like that, but it doesn't work:

courier_event_topic = app.topic(
    "courier_topic",
)

courier_table = app.Table(
    "courier_table",
    default=set,
    partitions=courier_event_topic.partitions,
)

Actual behavior

right now in code above courier_event_topic.partitions will be None

Versions

  • Python version: Python 3.10.6
  • Faust version: 0.10.14
  • Operating system: Ubuntu 22
  • Kafka version: 3.4
  • RocksDB version (if applicable)

This functionality does not currently exist in Faust. The best way I figure to go about doing this is to have a separate application track the topic metadata and propagate the number of partitions into some configuration file, which your Faust uses to determine the number of table partitions.