hibiken/asynqmon

Hang up for a long time and enter failed in about half an hour

tinnkai opened this issue · 4 comments

How to solve this situation? Thank you.
1643990450(1)

@tinnkai thank you for opening an issue!

Would you mind showing the output of

redis-cli ZRANGE asynq:{default}:deadlines 0 -1 WITHSCORES

@hibiken
image
It may be caused by the direct shutdown of the service while the queue is being processed,But what I expect can I keep the failure queue and then I can do it manually.
thank you

@hibiken
Sorry, I misunderstood. Processed includes the number of failed and successful, that is, entering failed will retry
image

@tinnkai Thanks for providing additional info!

I think initial confusion around task being stuck in "active" state is due to how the task recovery works currently when a worker crashes (or forced killed kill -9). Currently, asynq recovers task if its deadline has expired over 30s ago, but we have an open issue to improve this by recovering these orphaned tasks more quickly (hibiken/asynq#334).