Is there a way to assign the number of partitions on table dynamically relying on topic's number of partitions
dolshevsk opened this issue · 1 comments
Checklist
- I have included information about relevant versions
- I have verified that the issue persists when using the
master
branch of Faust.
Steps to reproduce
I want to know if there is a way to assign the number of partitions on table dynamically relying on topic's number of partitions. In my app I consume from external topics which doesn't belong to me which has different number of partitions in different environment and their number can be changed in future, but I do modify table under those topics and thus the number of partitions should be equal.
Expected behavior
I wanted to achieve it with something like that, but it doesn't work:
courier_event_topic = app.topic(
"courier_topic",
)
courier_table = app.Table(
"courier_table",
default=set,
partitions=courier_event_topic.partitions,
)
Actual behavior
right now in code above courier_event_topic.partitions
will be None
Versions
- Python version: Python 3.10.6
- Faust version: 0.10.14
- Operating system: Ubuntu 22
- Kafka version: 3.4
- RocksDB version (if applicable)
This functionality does not currently exist in Faust. The best way I figure to go about doing this is to have a separate application track the topic metadata and propagate the number of partitions into some configuration file, which your Faust uses to determine the number of table partitions.