Memory leak - celery threads don't go away

[jbothma]

Celery threads don't seem to go away after extract all which makes the machine run out of memory pretty quickly.

Have you seen this before?

[torfsen]

We ran into similar problems. The problem doesn't seem to be in ckanext-extractor, though. A workaround is to force a restart of the Celery workers after a certain number of tasks. Assuming you run Celery via paster, you can add the following to your CKAN's production.ini:

celeryd_max_tasks_per_child = 1

That will restart the workers after each task. If you have a lot of tasks you might need to increase that number since starting the workers takes some time.