Aiven-Open/gcs-connector-for-apache-kafka

[QUESTION]Save files in subfolder / Manage retention variables(time, size)

Closed this issue · 1 comments

It's possible to change the time that dump is made into the GCS bucket?
I see that every 3 minutes the files are saved and the only variable that I can change is file.max.records
But I thought in control of the file size(MB size) and dump at minimum time.

Exists a way to save files in subfolder into bucket? Instead of use an entire?

To save files in subfolders you can modify the file properties. E.g. this would save it in a folder structure like bucket:/folder/topic/year/month/day/

"file.name.prefix": "folder/",
"file.name.template": "{{topic}}/{{timestamp:unit=yyyy}}/{{timestamp:unit=MM}}/{{timestamp:unit=dd}}/{{partition}}-{{start_offset}}.json",