diamondio/better-queue-memory

Memory leak caused by empty batches fetched from store

Closed this issue · 1 comments

When fetching from the store, the queue pulls until it receives an empty batch. When an empty batch is pulled from the store, the MemoryStore creates a lock in the _running collection which is never released because the queue never attempts to release a lock from an empty batch.

Both of these checks return without ever releasing the lock for the empty task.
https://github.com/diamondio/better-queue/blob/master/lib/queue.js#L542-#L553

In long-running, high-use queues this could lead to resource leakage. For example, using the memory store, each empty lock is ~56B. After 1 Million cycles of this, this ends up being 56MB of lost memory.

I am posting this as an issue against the memory store instead of the queue code because both the Sqlite and MySQL stores seem to handle this properly themselves:
Sqlite: https://github.com/diamondio/better-queue-sqlite/blob/master/index.js#L141
MySQL: https://github.com/diamondio/better-queue-sql/blob/master/index.js#L44

Thanks for the fix!