Zbchaos worker has concurrency issues
Zelldon opened this issue · 3 comments
Right now we have implemented the zbchaos worker in a way that it just takes all the arguments from the experiment and calls the root command again with the given arguments.
Since we make use of globals in order to define flags this lead to heavy issues when running multiple commands after another (because properties/globals are not reset). Furthermore we can't execute multiple jobs at once, which is why we limited the max activation count to 1.
Quick fix: reset all properties in the worker
Long term solution: extract logic of commands and support parameters. Cobra Commands call the new logic/functions with the globals as parameters and the workers also call the new functions with parameters (instead of rely on globals) this makes it independent from the previous set of arguments from other actions/experiments.
For that we need a mapping of sub-commands to the functions, so we call the correct code.
When we implemented the long-term solution, we should be able also to handle more jobs concurrently. Needs to be verified since we use the kube-config
Long term solution: extract logic of commands and support parameters. Cobra Commands call the new logic/functions with the globals as parameters and the workers also call the new functions with parameters (instead of rely on globals) this makes it independent from the previous set of arguments from other actions/experiments.
For that we need a mapping of sub-commands to the functions, so we call the correct code.
Drawback is that we have to parse on our own the flags, which is really mae.
I could think of creating the commands and adding them to a root command in a separate function and maybe creating such commands and flags on each job worker handling 🤔