Stream loader not loading RabbitMQ queue
Opened this issue · 0 comments
roncewind commented
Is your feature request related to a problem? Please describe.
It'd be real swell if the loader understood when it wasn't actually pushing data into the queue. There seems to be a situation when the read thread found here:
fails, but the other threads continue to work. The monitor reports all is well, but no messages are being sent to RabbitMQ. Here's the sort of message found in the logs:
2022-02-24 18:38:43,457 senzing-50140129I Process-0-FilterQueueDictToJsonRabbitmqThread-3 is running.
2022-02-24 18:38:43,458 senzing-50140129I Process-0-MonitorThread-0 is running.
Exception in thread Process-0-FilterUrlJsonToDictQueueThread-0:
Traceback (most recent call last):
File "/usr/lib/python3.9/threading.py", line 954, in _bootstrap_inner
self.run()
File "/app/stream-producer.py", line 2118, in run
for message in self.read():
File "/app/stream-producer.py", line 1469, in read
for line in data:
File "/usr/lib/python3.9/http/client.py", line 659, in readline
result = self.fp.readline(limit)
File "/usr/lib/python3.9/socket.py", line 704, in readinto
return self._sock.recv_into(b)
File "/usr/lib/python3.9/ssl.py", line 1241, in recv_into
return self.read(nbytes, buffer)
File "/usr/lib/python3.9/ssl.py", line 1099, in read
return self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out
2022-02-24 18:48:43,962 senzing-50140127I Monitor: {"input_counter_interval": 0, "input_counter_line_number_in_file": 10001, "input_counter_rate_interval": 0, "input_counter_rate_total": 0, "input_counter_total": 0, "output_counter_interval": 0, "output_counter_line_number_in_file": 10001, "output_counter_rate_interval": 0, "output_counter_rate_total": 0, "output_counter_total": 0, "uptime": 605, "workers_active": 4, "workers_total": 5}
Looks like a single point of failure if the read thread times out or has other problems, the rest of the threads truck happily along.