Implement restarting the spark app every part
rom1504 opened this issue · 3 comments
rom1504 commented
or maybe every N parts
spark slows down for too long running apps
rom1504 commented
either find a way to close the context, either just run inside a process
rom1504 commented
definitely needed
starts at 1000s per part (25k wat)
a few hours later, 3000s per part
with per part session, it will stay at 1000s per part hence 3x faster
rom1504 commented
done