lensesio/stream-reactor

Source connector - read files from day partition

iamsamba opened this issue · 2 comments

iamsamba commented

Hi

I am using s3 source connector 2.8.4 to read parquet files s3 by year, month, day partition.

Connector connected successfully but 0 messages flushed.

flushing 0 outstanding messages for offset commit

Can someone let me know how do we define connector to flush messages from s3 parquet format by year, month, day partition?

connect.s3.kcql=insert into select * from PARTITIONBY _date.uuuu,_date.LL,_date.dd STOREAS parquet WITHPARTITIONER=Values
value.converter=io.confluent.connect.avro.AvroConverter
value.converter.schema.registry.url=

Thanks,

davidsloan commented

@iamsamba can you provide an example of your directory structure on S3?

The source doesn't support the PARTITIONBY keyword but instead uses a Partition Extractor that you define.

Please see the Partition Extraction section of our documentation for more details.

andrewstevenson commented

@iamsamba are you still having this issue? Did you try the latest release? Please reopen if the issue persists