malthe/pq

2.x roadmap

Opened this issue · 0 comments

malthe commented

For 2.x, PQ should support newer database drivers (see issue #71 which asks for psycopg 3.x support) and more importantly, embrace a new processing model akin to Azure Storage Queue.

In the new model, the queue operates in auto-commit mode with decoupled pull/complete logic.

When a queue item is checked out, a visibility timeout is persisted in the database. For the duration of this timeout, the worker is able to complete the work (either success or failure). If the timeout elapses, the queue item is available for a new worker to commit and at the same time, the original worker is no longer able to complete the item.

In practice, the worker receives a pop receipt when checking out the queue item and can complete the item using this receipt.

Tasks

Python

  • Update to psycopg 3.x
  • Type hints
  • Decoupled pull/complete logic

TypeScript

  • Implementation