Hang caused by circus and psutil using conflicting methods to identify a process

Line 460 in 4b8aaca

resulting_pid, status = os.waitpid(pid, os.WNOHANG)

psutil uses a combination of PID and process creation time to identify a process. This means that an attempt to terminate a process by circus can fail if psutil notices that the creation times do not match. In this case psutil does not send a terminate signal to the process, but instead returns a NoSuchProcess exception.

The loop in watcher.reap_process() just uses the PID to identify a process. This means that the loop containing os.waitpid(pid, os.WNOHANG) can run forever as it waits for a process to terminate that has not been told to terminate.

Although the probability of this event occurring may seem small, this problem can be triggered by the psutil bug described in giampaolo/psutil#2031