senzing-garage/stream-producer

Stream loader not loading RabbitMQ queue

Opened this issue · 0 comments

Is your feature request related to a problem? Please describe.
It'd be real swell if the loader understood when it wasn't actually pushing data into the queue. There seems to be a situation when the read thread found here:

https://github.com/Senzing/stream-producer/blob/240c580771a3c66f0c5da3e9045c334698b383cf/stream-producer.py#L2383-L2424

fails, but the other threads continue to work. The monitor reports all is well, but no messages are being sent to RabbitMQ. Here's the sort of message found in the logs:

2022-02-24 18:38:43,457 senzing-50140129I Process-0-FilterQueueDictToJsonRabbitmqThread-3 is running.
2022-02-24 18:38:43,458 senzing-50140129I Process-0-MonitorThread-0 is running.
Exception in thread Process-0-FilterUrlJsonToDictQueueThread-0:
Traceback (most recent call last):
  File "/usr/lib/python3.9/threading.py", line 954, in _bootstrap_inner
    self.run()
  File "/app/stream-producer.py", line 2118, in run
    for message in self.read():
  File "/app/stream-producer.py", line 1469, in read
    for line in data:
  File "/usr/lib/python3.9/http/client.py", line 659, in readline
    result = self.fp.readline(limit)
  File "/usr/lib/python3.9/socket.py", line 704, in readinto
    return self._sock.recv_into(b)
  File "/usr/lib/python3.9/ssl.py", line 1241, in recv_into
    return self.read(nbytes, buffer)
  File "/usr/lib/python3.9/ssl.py", line 1099, in read
    return self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out
2022-02-24 18:48:43,962 senzing-50140127I Monitor: {"input_counter_interval": 0, "input_counter_line_number_in_file": 10001, "input_counter_rate_interval": 0, "input_counter_rate_total": 0, "input_counter_total": 0, "output_counter_interval": 0, "output_counter_line_number_in_file": 10001, "output_counter_rate_interval": 0, "output_counter_rate_total": 0, "output_counter_total": 0, "uptime": 605, "workers_active": 4, "workers_total": 5}

Looks like a single point of failure if the read thread times out or has other problems, the rest of the threads truck happily along.