ClickHouse/spark-clickhouse-connector

Using connector with structured streaming

step4 opened this issue · 2 comments

step4 commented

I am using pyspark and this connector to write into a clickhouse cluster.
With normal batch dataframes everything seems to work.

Now I want to rewrite my job to use structured streaming to writes results from a delta table to the clickhouse cluster.

Is there a way to use the writeStream functionality or do I have to use the foreachBatch function?

Thanks for sharing your use case, currently, this connector does not support streaming writing, please use foreachBatch way.

step4 commented

Thanks for clarification.