Workers failed processing task

[mhamzahkhan]

Hello

I've just noticed an issue with my relay. The workers aren't processing any tasks, and are just expiring:

ERROR: 2023/02/28 12:41:38 worker.go:370 Failed processing task task_09138ca4-7b9c-4263-9f2d-8cd5d4aad69c. Error = activity ttl expired
ERROR: 2023/02/28 12:41:38 worker.go:370 Failed processing task task_ba76fe28-f96c-4b45-92d4-0ab095c62975. Error = activity ttl expired
ERROR: 2023/02/28 12:41:38 worker.go:370 Failed processing task task_adae57b1-c2d6-43e6-8a93-5d417e8f08bd. Error = activity ttl expired
ERROR: 2023/02/28 12:41:38 worker.go:370 Failed processing task task_c5faca59-6f4c-4bbf-99df-dbcc07e42137. Error = activity ttl expired
ERROR: 2023/02/28 12:41:38 worker.go:370 Failed processing task task_8817f889-57a0-47fb-8a0e-dadf019af666. Error = activity ttl expired
ERROR: 2023/02/28 12:41:38 worker.go:370 Failed processing task task_c2564b76-5994-4a04-a442-2010f42e88d5. Error = activity ttl expired
ERROR: 2023/02/28 12:41:38 worker.go:370 Failed processing task task_6de45395-b03d-435d-bacc-fd21342a6704. Error = activity ttl expired
ERROR: 2023/02/28 12:41:38 worker.go:370 Failed processing task task_4b60ebe0-9d31-4570-ade0-1640d579df92. Error = activity ttl expired
ERROR: 2023/02/28 12:41:38 worker.go:370 Failed processing task task_beba10e9-1c6e-4d90-a222-4dbe1b0fb2ac. Error = activity ttl expired
ERROR: 2023/02/28 12:41:38 worker.go:370 Failed processing task task_61e393e4-9ab8-4119-9a0e-d8275e971d8b. Error = activity ttl expired
ERROR: 2023/02/28 12:41:38 worker.go:370 Failed processing task task_2a0ab48a-642c-41dd-9581-cc8745f93106. Error = activity ttl expired
ERROR: 2023/02/28 12:41:38 worker.go:370 Failed processing task task_ebb48117-06c4-4a1f-be6c-6edf1ab08ccc. Error = activity ttl expired
ERROR: 2023/02/28 12:41:38 worker.go:370 Failed processing task task_cc1c2318-15dd-46ed-bde8-f0a8c9f8d3b3. Error = activity ttl expired
ERROR: 2023/02/28 12:41:38 worker.go:370 Failed processing task task_0556d110-aaa3-4a6f-8702-68c603150233. Error = activity ttl expired
ERROR: 2023/02/28 12:41:38 worker.go:370 Failed processing task task_ce18bf1e-7f45-4257-9ac1-b5273d7b8361. Error = activity ttl expired
ERROR: 2023/02/28 12:41:38 worker.go:370 Failed processing task task_d29b7e5d-4ee7-4343-b2d4-c81dad72c722. Error = activity ttl expired
ERROR: 2023/02/28 12:41:38 worker.go:370 Failed processing task task_957c5802-555e-4b23-908c-f6de9b163e87. Error = activity ttl expired
ERROR: 2023/02/28 12:41:38 worker.go:370 Failed processing task task_7e0c8e0a-5791-424c-9ef6-ca4a1fdf1c24. Error = activity ttl expired
ERROR: 2023/02/28 12:41:38 worker.go:370 Failed processing task task_a0ad5aa4-7ecb-4bc9-bf8a-28b2edf61d07. Error = activity ttl expired
ERROR: 2023/02/28 12:41:38 worker.go:370 Failed processing task task_b729a0d6-62d7-41fd-ac2b-f2ead0fd6a63. Error = activity ttl expired
ERROR: 2023/02/28 12:41:38 worker.go:370 Failed processing task task_a08359e8-c2ae-4a2d-841c-71fd1c897d97. Error = activity ttl expired
ERROR: 2023/02/28 12:41:38 worker.go:370 Failed processing task task_c0f385e7-a410-45ae-aa02-0d882ce018b2. Error = activity ttl expired
ERROR: 2023/02/28 12:41:38 worker.go:370 Failed processing task task_c164c46b-16af-4321-80c1-c4fe63d247ae. Error = activity ttl expired
ERROR: 2023/02/28 12:41:38 worker.go:370 Failed processing task task_82ca904c-7b18-48b7-ad14-8a4e219ec18a. Error = activity ttl expired
ERROR: 2023/02/28 12:41:38 worker.go:370 Failed processing task task_83932b37-45a0-4415-9184-f091f6f62549. Error = activity ttl expired
ERROR: 2023/02/28 12:41:38 worker.go:370 Failed processing task task_fd0c196a-29f4-4581-a5c7-d4a3143ba8da. Error = activity ttl expired

There aren't any other errors other than this. I'm not sure where to start debugging?
I'm running v1.2.5 at the moment.

[yukimochi]

This means "the activity was received more than 2 minutes ago".
I implement this to prevent that relay from sending too old status to subscribers.

To avoid them...

• Wait until the incoming call status cools down. (If it comes...)
• Power up your server specs or increase the number of workers.

To reset the task once...

1. Connect to Redis with redis-cli
2. Run del relay
   All previous tasks are then deleted, but the cause is not resolved and the same problem may occur again.

[mhamzahkhan]

Ah! I had some network problems with my cluster a few days ago, I think that's why the queue size increased must have increased dramatically.

It looks good now that I've manually cleared it! Thank you!