continuous crashes on pi 3b+

Question

continuous crashes on pi 3b+

JeremiahChurch opened this issue a year ago · 4 comments

I have read the README.md file and visited the Documentation to see if the information there helps.

Describe the bug
mqtt-io starts, connects successfully and will accept commands or publish status changes and then will crash and restart on about a 50 second cycle

Expected behavior
mqtt-io stays running

Error messages and traceback

2023-10-05 21:51:56 mqtt_io.server [INFO] Reconnecting to MQTT broker (infinite retries remaining)
2023-10-05 21:51:56 mqtt_io.server [INFO] Connecting to MQTT...
2023-10-05 21:51:56 mqtt_io.server [INFO] Connected to MQTT
2023-10-05 21:52:46 mqtt_io.server [ERROR] Exception in critical task:
Traceback (most recent call last):
  File "/usr/lib/python3.9/asyncio/locks.py", line 226, in wait
    await fut
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.9/asyncio/tasks.py", line 492, in wait_for
    fut.result()
asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/jchurch/ve/lib/python3.9/site-packages/asyncio_mqtt/client.py", line 217, in _wait_for
    return await asyncio.wait_for(*args, **kwargs)
  File "/usr/lib/python3.9/asyncio/tasks.py", line 494, in wait_for
    raise exceptions.TimeoutError() from exc
asyncio.exceptions.TimeoutError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/jchurch/ve/lib/python3.9/site-packages/mqtt_io/mqtt/asyncio_mqtt.py", line 32, in inner
    await func(*args, **kwargs)
  File "/home/jchurch/ve/lib/python3.9/site-packages/mqtt_io/mqtt/asyncio_mqtt.py", line 94, in publish
    await self._client.publish(
  File "/home/jchurch/ve/lib/python3.9/site-packages/asyncio_mqtt/client.py", line 136, in publish
    await self._wait_for(confirmation.wait(), timeout=timeout)
  File "/home/jchurch/ve/lib/python3.9/site-packages/asyncio_mqtt/client.py", line 219, in _wait_for
    raise MqttError('Operation timed out')
asyncio_mqtt.error.MqttError: Operation timed out

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/jchurch/ve/lib/python3.9/site-packages/mqtt_io/server.py", line 1218, in _main_loop
    await asyncio.gather(*self.critical_tasks)
  File "/home/jchurch/ve/lib/python3.9/site-packages/mqtt_io/server.py", line 1071, in _mqtt_keep_alive_loop
    await self.mqtt.publish(MQTTMessageSend(
  File "/home/jchurch/ve/lib/python3.9/site-packages/mqtt_io/mqtt/asyncio_mqtt.py", line 34, in inner
    raise MQTTException from exc
mqtt_io.mqtt.MQTTException

Config

mqtt:
  host: 10.10.10.10
  topic_prefix: aircomp
  user: mqtt
  password: 'reallygood'
  client_id: aircomp_pi
  ha_discovery:
    enabled: yes

# GPIO
gpio_modules:
  # Use the Raspberry Pi built-in GPIO
  - name: rpi
    module: raspberrypi

digital_outputs:
  # Pin 4, 22, 6, 26 - relays 1,2,3,4
  - name: primary_pump_enable
    module: rpi
    pin: 4
    ha_discovery:
      component: switch
      name: Air Comp Primary
  - name: secondary_pump_enable
    module: rpi
    pin: 22
  - name: drain_enable
    module: rpi
    pin: 6

Hardware

Platform: Raspberry Pi 3b+
Connected hardware: generic relay board connected to GPIO pins

System:

OS: Raspbian 11
Python version: 3.9.2
jchurch
Using a virtualenv?: yes

Additional context
didn't see any obvious points to check in the stacktrace or existing issues. happy to try anything and everything.

Love the library! <3

Answer 1 · 2023-10-06T12:26:35.000Z

Hi @JeremiahChurch,

who is restarting your mqtt-io? Are you using supervisor?
CancelledError says, that something killed your task...

Greetings
Benni

Answer 2 · 2023-10-06T12:39:49.000Z

who is restarting your mqtt-io? Are you using supervisor? CancelledError says, that something killed your task...

Apologies @BenjiU , I swear I had included that but I must have copy/pasted over it.

Using supervisor with a virtual env exactly as described in the docs. Only difference is config file path

Answer 3 · 2023-10-06T12:56:30.000Z

No Problem, may you please try without supervisor? Maybe supervisor does not recognice correct startup of mqtt-io and kills it.

Answer 4 · 2023-10-06T14:23:18.000Z

unfortunately similar outcome run directly

jchurch@aircompressor:~ $ sudo supervisorctl stop mqtt_io
mqtt_io: stopped
jchurch@aircompressor:~ $ /home/jchurch/ve/bin/python -m mqtt_io config.yml
2023-10-06 10:20:08 mqtt_io.server [INFO] Digital output 'primary_pump_enable' current value is True (raw: True)
2023-10-06 10:20:09 mqtt_io.server [INFO] Digital output 'secondary_pump_enable' current value is False (raw: False)
2023-10-06 10:20:10 mqtt_io.server [INFO] Digital output 'drain_enable' current value is False (raw: False)
2023-10-06 10:20:10 mqtt_io.server [INFO] Digital output 'unused_relay' current value is False (raw: False)
2023-10-06 10:20:10 mqtt_io.server [INFO] Connecting to MQTT...
2023-10-06 10:20:10 mqtt_io.server [INFO] Connected to MQTT
2023-10-06 10:20:10 mqtt_io.server [INFO] Subscribed to topic: 'aircomp/output/primary_pump_enable/set'
2023-10-06 10:20:10 mqtt_io.server [INFO] Subscribed to topic: 'aircomp/output/primary_pump_enable/set_on_ms'
2023-10-06 10:20:10 mqtt_io.server [INFO] Subscribed to topic: 'aircomp/output/primary_pump_enable/set_off_ms'
2023-10-06 10:20:10 mqtt_io.server [INFO] Subscribed to topic: 'aircomp/output/drain_enable/set'
2023-10-06 10:20:10 mqtt_io.server [INFO] Subscribed to topic: 'aircomp/output/drain_enable/set_on_ms'
2023-10-06 10:20:10 mqtt_io.server [INFO] Subscribed to topic: 'aircomp/output/drain_enable/set_off_ms'
2023-10-06 10:20:10 mqtt_io.server [INFO] Subscribed to topic: 'aircomp/output/secondary_pump_enable/set'
2023-10-06 10:20:10 mqtt_io.server [INFO] Subscribed to topic: 'aircomp/output/secondary_pump_enable/set_on_ms'
2023-10-06 10:20:10 mqtt_io.server [INFO] Subscribed to topic: 'aircomp/output/secondary_pump_enable/set_off_ms'
2023-10-06 10:20:10 mqtt_io.server [INFO] Subscribed to topic: 'aircomp/output/unused_relay/set'
2023-10-06 10:20:10 mqtt_io.server [INFO] Subscribed to topic: 'aircomp/output/unused_relay/set_on_ms'
2023-10-06 10:20:10 mqtt_io.server [INFO] Subscribed to topic: 'aircomp/output/unused_relay/set_off_ms'
2023-10-06 10:21:00 mqtt_io.server [ERROR] Exception in critical task:
Traceback (most recent call last):
  File "/usr/lib/python3.9/asyncio/locks.py", line 226, in wait
    await fut
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.9/asyncio/tasks.py", line 492, in wait_for
    fut.result()
asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/jchurch/ve/lib/python3.9/site-packages/asyncio_mqtt/client.py", line 217, in _wait_for
    return await asyncio.wait_for(*args, **kwargs)
  File "/usr/lib/python3.9/asyncio/tasks.py", line 494, in wait_for
    raise exceptions.TimeoutError() from exc
asyncio.exceptions.TimeoutError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/jchurch/ve/lib/python3.9/site-packages/mqtt_io/mqtt/asyncio_mqtt.py", line 32, in inner
    await func(*args, **kwargs)
  File "/home/jchurch/ve/lib/python3.9/site-packages/mqtt_io/mqtt/asyncio_mqtt.py", line 94, in publish
    await self._client.publish(
  File "/home/jchurch/ve/lib/python3.9/site-packages/asyncio_mqtt/client.py", line 136, in publish
    await self._wait_for(confirmation.wait(), timeout=timeout)
  File "/home/jchurch/ve/lib/python3.9/site-packages/asyncio_mqtt/client.py", line 219, in _wait_for
    raise MqttError('Operation timed out')
asyncio_mqtt.error.MqttError: Operation timed out

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/jchurch/ve/lib/python3.9/site-packages/mqtt_io/server.py", line 1218, in _main_loop
    await asyncio.gather(*self.critical_tasks)
  File "/home/jchurch/ve/lib/python3.9/site-packages/mqtt_io/server.py", line 1071, in _mqtt_keep_alive_loop
    await self.mqtt.publish(MQTTMessageSend(
  File "/home/jchurch/ve/lib/python3.9/site-packages/mqtt_io/mqtt/asyncio_mqtt.py", line 34, in inner
    raise MQTTException from exc
mqtt_io.mqtt.MQTTException
2023-10-06 10:21:02 mqtt_io.server [INFO] Reconnecting to MQTT broker (infinite retries remaining)
2023-10-06 10:21:02 mqtt_io.server [INFO] Connecting to MQTT...
2023-10-06 10:21:02 mqtt_io.server [INFO] Connected to MQTT