YotpoLtd/metorikku

Is metorikku support hudi incremental pull as input?

SpeedxPz opened this issue · 1 comments

I'm new at data warehouse and currently using Metorikku for streaming CDC from Kafka and sink into the data lake as Hudi

I have to do the ETL process after that
Can Metorikku do incremental pull from the Hudi?

Thanks

In general you have a lot of metadata fields to indicate the commit time/id of each row in the hudi table.
So you could WHERE these to get the last commit or something (you will need to know what you are looking for though)