how to execute metorikku in a loop from a single spark-submit?
Opened this issue · 1 comments
tooptoop4 commented
use-case: near-real time sync from MSSQL db to s3.
I know I can do a while loop calling spark-submit each time but this will be slow as JVM needs to startup each time. Is there a way to make metorikku keep on looping through the same input/metric.yaml within a single spark-submit?
note: i know should have kafka..etc but not an option at this stage
lyogev commented
We don't have such a feature, but it's a cool idea to create fake micro batches in metorikku to stream on non streaming sources