Unstable connection to redis on Toolforge

[wetneb]

[2024-05-22 13:39:42,965: WARNING/MainProcess] consumer: Connection to broker lost. Trying to re-establish the connection...
Traceback (most recent call last):
  File "/data/project/editgroups/www/python/venv/lib/python3.11/site-packages/redis/connection.py", line 706, in send_packed_command
    sendall(self._sock, item)
  File "/data/project/editgroups/www/python/venv/lib/python3.11/site-packages/redis/_compat.py", line 9, in sendall
    return sock.sendall(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TimeoutError: [Errno 110] Connection timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/data/project/editgroups/www/python/venv/lib/python3.11/site-packages/celery/worker/consumer/consumer.py", line 340, in start
    blueprint.start(self)
  File "/data/project/editgroups/www/python/venv/lib/python3.11/site-packages/celery/bootsteps.py", line 116, in start
    step.start(parent)
  File "/data/project/editgroups/www/python/venv/lib/python3.11/site-packages/celery/worker/consumer/consumer.py", line 746, in start
    c.loop(*c.loop_args())
  File "/data/project/editgroups/www/python/venv/lib/python3.11/site-packages/celery/worker/loops.py", line 97, in asynloop
    next(loop)
  File "/data/project/editgroups/www/python/venv/lib/python3.11/site-packages/kombu/asynchronous/hub.py", line 308, in create_loop
    poll_timeout = fire_timers(propagate=propagate) if scheduled else 1
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/project/editgroups/www/python/venv/lib/python3.11/site-packages/kombu/asynchronous/hub.py", line 149, in fire_timers
    entry()
  File "/data/project/editgroups/www/python/venv/lib/python3.11/site-packages/kombu/asynchronous/timer.py", line 70, in __call__
    return self.fun(*self.args, **self.kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/project/editgroups/www/python/venv/lib/python3.11/site-packages/kombu/asynchronous/timer.py", line 137, in _reschedules
    return fun(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^
  File "/data/project/editgroups/www/python/venv/lib/python3.11/site-packages/kombu/transport/redis.py", line 564, in maybe_check_subclient_health
    client.check_health()
  File "/data/project/editgroups/www/python/venv/lib/python3.11/site-packages/redis/client.py", line 3521, in check_health
    conn.send_command('PING', self.HEALTH_CHECK_MESSAGE,
  File "/data/project/editgroups/www/python/venv/lib/python3.11/site-packages/redis/connection.py", line 725, in send_command
    self.send_packed_command(self.pack_command(*args),
  File "/data/project/editgroups/www/python/venv/lib/python3.11/site-packages/redis/connection.py", line 709, in send_packed_command
    raise TimeoutError("Timeout writing to socket")
redis.exceptions.TimeoutError: Timeout writing to socket

[wetneb]

Fixed by migrating to our own redis instance.