Can I set my custom id for spark structured streaming query?
Mvpanswer7 opened this issue · 1 comments
Mvpanswer7 commented
In spark structured streaming, we can define a stream query like :
var writer: DataStreamWriter[Row] = DF.writeStream
val query = writer.trigger(Trigger.ProcessingTime(option("duration").toInt, TimeUnit.SECONDS)).start()
Then we can get id by query.id
and get runId by query.RunId
.
My problem is can I use my own id for query.id to such query, I haven't found any effective method like setId to set an id to query.
jaceklaskowski commented
Please ask this question at StackOverflow.