Restart a consumer after a certain number of tasks
mindojo-victor opened this issue · 11 comments
Is it easy to implement something like http://docs.celeryproject.org/en/latest/userguide/configuration.html#std:setting-worker_max_tasks_per_child ?
Or if there was a mechanism to restart the consumer manually ( #251 ) it should be easy to count tasks by myself and decide whether I need to restart the consumer because of leaking memory.
Right now there is not, but I don't see why we should implement a restart signal at the very least. SIGHUP for instance?
I'm suggesting using SIGHUP to trigger a restart. I'm not too sure how to restart processes with Python from within the process itself... is it sufficient to restart the workers or should the whole consumer process get restarted, for example? Would I use the os.exec family to start the new process?
Looks like I want https://docs.python.org/3/library/os.html#os.execl
If I understood correctly, if the master process receives SIGHUP, it restart itself with all the workers. If a worker receives SIGHUP, only it restarts. I am interested in the second case, because other workers could be ok.
Wanna try out 1ffb3dd ?
Will try this later. Thank you!
I did not implement this on a worker-by-worker basis. It applies to the consumer as a whole.
If you're running the worker health checker, it should be possible to just kill the worker process using the standard SIGTERM and huey consumer will automatically restart a new worker to take it's place?
This should work. If health checker always running?
Looks like it is. This means that all I need is there in place already. I can count number of tasks and raise KeyboardInterrupt when I think the worker should restart. It will die and the consumer will raise another worker in its place?