japila-books/spark-structured-streaming-internals

Can I set my custom id for spark structured streaming query?

Mvpanswer7 opened this issue · 1 comments

In spark structured streaming, we can define a stream query like :

var writer: DataStreamWriter[Row] = DF.writeStream

val query = writer.trigger(Trigger.ProcessingTime(option("duration").toInt, TimeUnit.SECONDS)).start()

Then we can get id by query.id and get runId by query.RunId.

My problem is can I use my own id for query.id to such query, I haven't found any effective method like setId to set an id to query.

Please ask this question at StackOverflow.