Set up mechanism for triggering the batch pipeline
rviscomi opened this issue · 1 comments
rviscomi commented
If we're doing away with streaming pipelines, the batch jobs need to be triggered when the crawl is complete.
giancarloaf commented
This includes leveraging GCP workflows + dataflow flex templates for triggering the whole process.