Cluster always self destructs after 30 minutes.
torkjel opened this issue · 7 comments
The spydra/init-actions/self-destruct.sh script will install the self destruct cron job on both master and the 0th node. However only the master receives the heartbeat updates, so the 0th worker will always kill the cluster when the collector timeout is reached.
Hi torkjel,
Thanks for reporting. I ran into a similar issue recently. Could you confirm wether this issue is appearing for you if you fix the Dataproc image version to 1.1 by adding the following configuration to the cluster section of your Spydra configuration file: "image-version": "1.1.24".
Hi Steffen,
I can't see how the image version would matter... There are two self destruct cron jobs running, but spydra only sends heartbeats to the one on master. The DynamicSubmitter.Heartbeater code is not aware that there exist a second self destruct job on the 0th worker.
Yeah I agree with you it looks like it is broken. Interestingly, it doesn't affect our jobs though using the 1.1.24 image but that might be cause by something else. We are looking into it.
Maybe you're running with a large enough timeout so you don't encounter it that much? That's my workaround for now. Thanks for looking into it!
@torkjel I just merged a fix for the issue, there's no release yet but feel free to test it and let us know if it helps.
The caveat of the solution is that total size of the metadata for project and/or instance is limited by 512Kb, so in theory you can reach that limit by spawning a lot of clusters. I believe it's still a good enough solution for now and should work for most workloads.
Sorry for not getting back to issue this sooner. 0.3.15 works exactly as expected in this regard. Thanks!