Aiven-Open/gcs-connector-for-apache-kafka

Single connector multiple topics corresponding one prefix/topic

Closed this issue · 2 comments

Hi
I am testing out the connector for our use-case,
i have multiple topics (A,B,C lets say) created for the same source. I want to push this data to the same bucket, but with a different prefix, for eg:
topic A goes into bucket/topic_A/
topic B goes into bucket/topic_B/
topic C goes into bucket/topic_C/

I can do this by creating three instance of the connector but since there is only 1 variable here, I was wondering if i can use the same connector instance to process data from all topics and push them accordingly into the right prefix based on which topic the data is coming from.

Is there a provision for this. I have tried to read the docs but couldn't find something that could satisfy my requirement.
Please let me know if i have missed on something.

Thanks!

i found a way to deal with this so just leaving it for everyone here in case they need
i ignored the prefix and used
file.name.template: "{{topic}}/year={{timestamp:unit=yyyy}}/month={{timestamp:unit=MM}}/day={{timestamp:unit=dd}}/hour={{timestamp:unit=HH}}/{{topic}}-{{partition:padding=false}}-{{start_offset:padding=false}}"

Yes, the file.name.template is the way!