Using connector with structured streaming
step4 opened this issue · 2 comments
step4 commented
I am using pyspark and this connector to write into a clickhouse cluster.
With normal batch dataframes everything seems to work.
Now I want to rewrite my job to use structured streaming to writes results from a delta table to the clickhouse cluster.
Is there a way to use the writeStream
functionality or do I have to use the foreachBatch
function?
pan3793 commented
Thanks for sharing your use case, currently, this connector does not support streaming writing, please use foreachBatch
way.
step4 commented
Thanks for clarification.