Is metorikku support hudi incremental pull as input?
SpeedxPz opened this issue · 1 comments
SpeedxPz commented
I'm new at data warehouse and currently using Metorikku for streaming CDC from Kafka and sink into the data lake as Hudi
I have to do the ETL process after that
Can Metorikku do incremental pull from the Hudi?
Thanks
lyogev commented
In general you have a lot of metadata fields to indicate the commit time/id of each row in the hudi table.
So you could WHERE these to get the last commit or something (you will need to know what you are looking for though)