stenotype sometimes doesn't shut down when it gets SIGINT/SIGTERM signals

Question

stenotype sometimes doesn't shut down when it gets SIGINT/SIGTERM signals

gconnell opened this issue 10 years ago · 3 comments

Not sure what's going on or how to debug, but sometimes stenographer will not shut down when it gets a SIGTERM. It registers the signal (logs from Handle*), but I think that when it joins one of the threads (not sure which), they don't ever finish.

Answer 1 · 2014-12-10T00:20:58.000Z

This appears to be stuck waiting for index writer threads to finish, but they never do. Those threads appear to be blocking on the producer/consumer queue waiting for a new file to write... they didn't get the NULL we sent? Might try to rewrite ProducerConsumerQueue with an explicit Close() to shut it down.

Answer 2 · 2014-12-10T01:01:20.000Z

Tracked this down, I think, to thread::join on the RunThread threads not returning.

Answer 3 · 2014-12-12T15:53:52.000Z

So the final determination was a deadlock in the packet reading/writing threads, which caused a deadlock in the PacketsV3 destructor. This means that the main thread was stuck waiting on the thread::joins for RunThread, but that the final LOG message in RunThread had already been written. So the "stuck index writers" were a red herring... the main thread hadn't gotten to the point where it could write its NULLs to stop them.