newfront/hitchhikers_guide_to_deltalake_streaming

Finish Intro to DL Streaming

Closed this issue a year ago · 0 comments

newfront commented a year ago

Finishing up the flow to use trigger.once and reduce the size of the trigger to 60 records.

introduce how to use triggering to throttle a readStream
using trigger(once=...) in pyspark to do one stateful batch
view the checkpoints (introduce checkpoints)
introduce maxFilesPerTrigger and other properties