Add redis cluster broker and result-backend
steigerung opened this issue · 7 comments
Adding the Redis Cluster broker and result-backend out of the box would be nice advantage, relative to alternative task queues.
But wouldn't it be redundant?
If this idea seems relevant to you, there are some implementation issues.
Using redis-py there is no problem with developing a result-backend, but there are some problems with the broker, more specifically redis.asyncio.RedisCluster client doesn't support pubsub, so it seems the only way to implement AsyncBroker correctly is to initialize redis.asyncio.RedisCluster and subscribe on tasks kicks channel in any master-node through redis.asyncio.Redis, as long as it is alive and switch to another if the current master-node is not working.
@chandr-andr up
@steigerung Hi!
I apologize for the long feedback.
The proposal to use a Redis Cluster is extremely interesting.
I will try to make an MVP that can implement your idea.
I'm interested in this. As I see it, implementing support for redis cluster in result backend should be trivial, because it doesn't use PUB/SUB. It only uses SET, EXISTS, GET and GETDEL which are all supported by redis cluster in redis-py.
Even if we only did this, it would still be an improvement, because there are multiple brokers available for taskiq, but only one result backend (at least that's what the official taskiq docs suggests).
I'm willing to implement changes for result backend to support cluster mode. Expect a PR in a few days :-).
Hi @s3riusI
Saw that you accepted the pool request, when can I expect a new version of the library? I'm really looking forward to these changes
Hi! I guess we can safely release it now. Because these changes are backward compatible.
Are there any plans to create a ListQueueBroker and a PubSubBroker with cluster support?
Sure. I'm not certain about PubSub support. Because in cluster mode it's a bit tricky. But anyway it sounds like a good idea.