[CRITICAL] Consumer encountered an unexpected error, IndexError: pop from empty list
mhdzumair opened this issue · 2 comments
mhdzumair commented
Issues
GitHub issues are for bugs. If you have questions, please ask them on the mailing list.
Checklist
- Does your title concisely summarize the problem?
- Did you include a minimal, reproducible example?
- What OS are you using?
- What version of Dramatiq are you using?
- What did you do?
- What did you expect would happen?
- What happened?
What OS are you using?
Initially failed in oracle Linux instances in k8s, Then tested with Debian. The issue is recently occurred in Oracle Redis cluster. Its recent issue, not from beginning.
What version of Dramatiq are you using?
1.16.0
What did you do?
Running dramatiq api.task --process 1, I noticed an issue on prod today. I tried to replicate it from local with the same Redis cluster and got the same result. After that, I tried deleting all keys with dramatiq* in Redis, but the issue persisted. Finally, I used another Redis container, and the issue was resolved.
What did you expect would happen?
Work as expected.
What happened?
Error log,
[2024-02-23 07:53:14,316] [PID 29212] [Thread-5] [dramatiq.worker.ConsumerThread(default.DQ)] [CRITICAL] Consumer encountered an unexpected error.
Traceback (most recent call last):
File "/home/dedsec/.local/share/virtualenvs/mediafusion-qrlDKCVP/lib/python3.11/site-packages/dramatiq/brokers/redis.py", line 344, in __next__
data = self.message_cache.pop(0)
^^^^^^^^^^^^^^^^^^^^^^^^^
IndexError: pop from empty list
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/dedsec/.local/share/virtualenvs/mediafusion-qrlDKCVP/lib/python3.11/site-packages/redis/commands/core.py", line 5931, in __call__
return client.evalsha(self.sha, len(keys), *args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/dedsec/.local/share/virtualenvs/mediafusion-qrlDKCVP/lib/python3.11/site-packages/redis/commands/core.py", line 5307, in evalsha
return self._evalsha("EVALSHA", sha, numkeys, *keys_and_args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/dedsec/.local/share/virtualenvs/mediafusion-qrlDKCVP/lib/python3.11/site-packages/redis/commands/core.py", line 5291, in _evalsha
return self.execute_command(command, sha, numkeys, *keys_and_args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/dedsec/.local/share/virtualenvs/mediafusion-qrlDKCVP/lib/python3.11/site-packages/redis/client.py", line 536, in execute_command
return conn.retry.call_with_retry(
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/dedsec/.local/share/virtualenvs/mediafusion-qrlDKCVP/lib/python3.11/site-packages/redis/retry.py", line 46, in call_with_retry
return do()
^^^^
File "/home/dedsec/.local/share/virtualenvs/mediafusion-qrlDKCVP/lib/python3.11/site-packages/redis/client.py", line 537, in <lambda>
lambda: self._send_command_parse_response(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/dedsec/.local/share/virtualenvs/mediafusion-qrlDKCVP/lib/python3.11/site-packages/redis/client.py", line 513, in _send_command_parse_response
return self.parse_response(conn, command_name, **options)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/dedsec/.local/share/virtualenvs/mediafusion-qrlDKCVP/lib/python3.11/site-packages/redis/client.py", line 553, in parse_response
response = connection.read_response()
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/dedsec/.local/share/virtualenvs/mediafusion-qrlDKCVP/lib/python3.11/site-packages/redis/connection.py", line 524, in read_response
raise response
redis.exceptions.NoScriptError: No matching script. Please use EVAL.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/dedsec/.local/share/virtualenvs/mediafusion-qrlDKCVP/lib/python3.11/site-packages/dramatiq/worker.py", line 264, in run
for message in self.consumer:
File "/home/dedsec/.local/share/virtualenvs/mediafusion-qrlDKCVP/lib/python3.11/site-packages/dramatiq/brokers/redis.py", line 363, in __next__
self.message_cache = messages = self.broker.do_fetch(
^^^^^^^^^^^^^^^^^^^^^
File "/home/dedsec/.local/share/virtualenvs/mediafusion-qrlDKCVP/lib/python3.11/site-packages/dramatiq/brokers/redis.py", line 278, in do_dispatch
self._max_unpack_size(),
^^^^^^^^^^^^^^^^^^^^^^^
File "/home/dedsec/.local/share/virtualenvs/mediafusion-qrlDKCVP/lib/python3.11/site-packages/dramatiq/brokers/redis.py", line 255, in _max_unpack_size
cls._max_unpack_size_val = DEFAULT_LUA_MAX_STACK or self.scripts["maxstack"]()
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/dedsec/.local/share/virtualenvs/mediafusion-qrlDKCVP/lib/python3.11/site-packages/redis/commands/core.py", line 5936, in __call__
self.sha = client.script_load(self.script)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/dedsec/.local/share/virtualenvs/mediafusion-qrlDKCVP/lib/python3.11/site-packages/redis/commands/core.py", line 5377, in script_load
return self.execute_command("SCRIPT LOAD", script)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/dedsec/.local/share/virtualenvs/mediafusion-qrlDKCVP/lib/python3.11/site-packages/redis/client.py", line 536, in execute_command
return conn.retry.call_with_retry(
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/dedsec/.local/share/virtualenvs/mediafusion-qrlDKCVP/lib/python3.11/site-packages/redis/retry.py", line 46, in call_with_retry
return do()
^^^^
File "/home/dedsec/.local/share/virtualenvs/mediafusion-qrlDKCVP/lib/python3.11/site-packages/redis/client.py", line 537, in <lambda>
lambda: self._send_command_parse_response(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/dedsec/.local/share/virtualenvs/mediafusion-qrlDKCVP/lib/python3.11/site-packages/redis/client.py", line 513, in _send_command_parse_response
return self.parse_response(conn, command_name, **options)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/dedsec/.local/share/virtualenvs/mediafusion-qrlDKCVP/lib/python3.11/site-packages/redis/client.py", line 553, in parse_response
response = connection.read_response()
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/dedsec/.local/share/virtualenvs/mediafusion-qrlDKCVP/lib/python3.11/site-packages/redis/connection.py", line 524, in read_response
raise response
redis.exceptions.ResponseError: unknown command 'SCRIPT', with args beginning with: 'LOAD' '-- This file is a part of Dramatiq. -- -- Copyright (C) 2020 CLEARTYPE SRL <bogdan@cleartype.io> -- -- Dramatiq is free s'
[2024-02-23 07:53:14,319] [PID 29212] [Thread-5] [dramatiq.worker.ConsumerThread(default.DQ)] [INFO] Restarting consumer in 3.00 seconds.
mhdzumair commented
I even tried to flushallon that Redis cluster and rerun the dramatic worker, still the same issue.
mhdzumair commented
I think it might be an issue with the subscription of the cluster. Closing as not needed.