Que v1 compatibility, redesign
bgentry opened this issue · 2 comments
Que has a v1 beta which makes some big changes that should lead to significant improvements in throughput and connection overhead. These are the highlights:
- Que's implementation has been changed from one in which worker threads hold their own PG connections and lock their own jobs to one in which a single thread (and PG connection) locks jobs through LISTEN/NOTIFY and batch polling, and passes jobs along to worker threads. This has many benefits, including:
- Jobs queued for immediate processing can be actively distributed to workers with LISTEN/NOTIFY, which is more efficient than having workers repeatedly poll for new jobs.
- When polling is necessary (to pick up jobs that are scheduled for the future or that need to be retried due to errors), jobs can be locked and fetched in batches, rather than one at a time.
- Individual workers no longer need to monopolize their own (usually idle) connections while working jobs, so Ruby processes will require fewer Postgres connections.
- PgBouncer or another external connection pool can be used for workers' connections (though not for the connection used to lock and listen for jobs).
More details are in the changelog.
I've upgraded my own project to use the beta version in Ruby and it's working great thus far. Compatibility would require a redesign of this library, but that concurrency story should actually be pretty easy to build in Go. I hope to have the time to work on that soon, but if somebody beats me to it I won't be upset 😄.
Quite a bit has changed in the Go SQL ecosystem since que-go was originally written so there could probably be a lot of different design decisions made.
Hi @bgentry, I'd like to volunteer !
What's the status of this?