From Kue to featureless-job-queue: increased performance by 3

[Neamar]

This is a quick writeup of the changes I got on a sample application when migrating from Kue to featureless-job-queue.

Please keep in mind that this library is only useful for non-fancy workloads: no priorities, no retries... pretty much nothing except a simple distributed FIFO.

Job creation

Crating 5000 jobs with Kue used to take upward of 30s, even with a high concurrency (200 simultaneous commands sent to Redis).

Since featureless-job-queue has an option to batch jobs and doesn't require to set 5 Redis keys / jobs, this is already a big improvement:

Note that depending on your workload, tuning options.cargoConcurrency could yield even better results. For now, I've sticked with the default value of 200.

Job processing

For high-throughput operation (concurrency > 5), Kue was spending a lot of time getting the job id and then getting job data from Redis. For simple jobs, this sometimes meant that half of the workers were asleep because Kue wasn't able to get jobs fast enough.

featureless-job-queue only use BLPOP to retrieve the job data. Additionally, there is an internal buffer maintained to ensure that if many workers finish at the same time, they won't be starved waiting for a Redis roundtrip. This buffer default to 10% of the current job concurrency.

Here is the number of tasks processed with Kue (before) and featureless-job-queue (after):

Connections to Redis

For two different processes, Kue was using 7 connections to Redis
featureless-job-queue only uses 2 connections, plus one more if you want to push tasks from the same worker -- so 3 max.

Redis operation

Even though Redis is fast, this is not a reason to send more commands than you need. Here is another before / after in terms of number of operations done in Redis:

Memory usage

Memory usage for the workers was divided by two when moving from Kue to featureless-job-queue:

I'm not really sure how to explain this one.

What's the point?

This is just a reminder that you don't need a bazooka to kill the proverbial ant.
Kue is excellent for robust use cases, but anything simpler can benefit from a more direct approach.