clean up crashed scenarios
Closed this issue · 3 comments
Just looking at htop
inside the rpc-0 container im seeing processes listed from scenarios that are not showing as "active"
. I think we need to .join()
those scenario threads when they either crash or finish.
I imagine this is the logging thread that's creating the stranded processes.
I'm going to add a handle to the logging thread onto the process dict, and join it in scenarios_stop
I have not had a chance to test this, but it gives an idea of what I'm thinking in terms of cleaning up the logging thread: main...mplsgrant:warnet:2024-07-process-context
I'm actually not able to faithfully reproduce this anymore. Starting scenarios with --invalid_options
or using run-file
with a file with syntax errors or using scenarios stop
, all threads disappear appropriately in htop. Going to close for now until I see it again