Error while fetching scheduled tasks from redis cluster
Closed this issue · 2 comments
tokikanno commented
while using taskiq-redis schedule source on redis cluster with more than 1 node, it will refuse to run while executing mget command in the get_schedules() function
I think it's because not all scheduled tasks are stored on the same key slot, so the mget refused to fetch.
File "/usr/local/lib/python3.11/site-packages/taskiq/cli/scheduler/run.py", line 45, in get_schedules
return await source.get_schedules()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/taskiq_redis/schedule_source.py", line 166, in get_schedules
schedules.extend(await self.redis.mget(buffer)) # type: ignore[attr-defined]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
# truncated ....
File "/usr/local/lib/python3.11/site-packages/redis/asyncio/cluster.py", line 593, in _determine_nodes
await self._determine_slot(command, *args),
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/redis/asyncio/cluster.py", line 643, in _determine_slot
raise RedisClusterException(
redis.exceptions.RedisClusterException: MGET - all keys must map to the same key slot
s3rius commented
Yes, you're right. Will fix soon.
tokikanno commented
Some extra info:
- This issue will happen even in a single node redis cluster
- This issue can be bypassed using manual hash tag assignment skill, but may also cause hot spot if having enought amounts of scheduled tasks
schedule_source: RedisClusterScheduleSource = RedisClusterScheduleSource(
url=_redis_url, prefix="{schedule}"
)