Lost communication with every zigbee device on conbee 2

Question

Lost communication with every zigbee device on conbee 2

ahaverty opened this issue 2 years ago · 26 comments

I've been running ZHA on a conbee 2 via HA for over a year now, but last night every zigbee device lost connectivity.

Lights
Motion
Switches
All from various manufacturers from Ikea, unk, lumi, ledvance, lidl etc

There was no update done, or any physical changes made to my HA or ZHA.

I've tried rebooting ZHA, restarting home assistant, disconnecting/reconneting my conbee usb.

The only thing that seems to fix for me is individually setting each device back into pairing mode, and adding via Zha again (It remembers the device name/id thankfully).

In terms of related logs, I see:

2022-10-19 08:05:25.863 WARNING (MainThread) [zigpy_deconz.zigbee.application] Unexpected transmit confirm for request id 196, Status: TXStatus.NWK_ROUTE_DISCOVERY_FAILED
2022-10-19 08:05:31.784 WARNING (MainThread) [homeassistant.components.zha.core.channels.base] [0x5950:1:0x0006]: async_initialize: all attempts have failed: [DeliveryError('Failed to deliver packet: <TXStatus.NWK_ROUTE_DISCOVERY_FAILED: 208>'), DeliveryError('Failed to deliver packet: <TXStatus.NWK_ROUTE_DISCOVERY_FAILED: 208>'), DeliveryError('Failed to deliver packet: <TXStatus.NWK_ROUTE_DISCOVERY_FAILED: 208>'), DeliveryError('Failed to deliver packet: <TXStatus.NWK_ROUTE_DISCOVERY_FAILED: 208>')]
2022-10-19 08:05:31.799 WARNING (MainThread) [homeassistant.components.zha.core.channels.base] [0x7E37:1:0x0006]: async_initialize: all attempts have failed: [DeliveryError('Failed to deliver packet: <TXStatus.NWK_ROUTE_DISCOVERY_FAILED: 208>'), DeliveryError('Failed to deliver packet: <TXStatus.NWK_ROUTE_DISCOVERY_FAILED: 208>'), DeliveryError('Failed to deliver packet: <TXStatus.NWK_ROUTE_DISCOVERY_FAILED: 208>'), DeliveryError('Failed to deliver packet: <TXStatus.NWK_ROUTE_DISCOVERY_FAILED: 208>')]
2022-10-19 08:05:31.866 WARNING (MainThread) [homeassistant.components.zha.core.channels.base] [0x9039:1:0x0702]: async_initialize: all attempts have failed: [DeliveryError('Failed to deliver packet: <TXStatus.NWK_ROUTE_DISCOVERY_FAILED: 208>'), DeliveryError('Failed to deliver packet: <TXStatus.NWK_ROUTE_DISCOVERY_FAILED: 208>'), DeliveryError('Failed to deliver packet: <TXStatus.NWK_ROUTE_DISCOVERY_FAILED: 208>'), DeliveryError('Failed to deliver packet: <TXStatus.NWK_ROUTE_DISCOVERY_FAILED: 208>')]
2022-10-19 08:05:31.951 WARNING (MainThread) [homeassistant.components.zha.core.channels.base] [0xD15B:1:0x0006]: async_initialize: all attempts have failed: [DeliveryError('Failed to deliver packet: <TXStatus.NWK_ROUTE_DISCOVERY_FAILED: 208>'), DeliveryError('Failed to deliver packet: <TXStatus.NWK_ROUTE_DISCOVERY_FAILED: 208>'), DeliveryError('Failed to deliver packet: <TXStatus.NWK_ROUTE_DISCOVERY_FAILED: 208>'), DeliveryError('Failed to deliver packet: <TXStatus.NWK_ROUTE_DISCOVERY_FAILED: 208>')]
2022-10-19 08:05:31.960 WARNING (MainThread) [homeassistant.components.zha.core.channels.base] [0xD15B:1:0x0008]: async_initialize: all attempts have failed: [DeliveryError('Failed to deliver packet: <TXStatus.NWK_ROUTE_DISCOVERY_FAILED: 208>'), ControllerException('duplicate 4 TSN'), DeliveryError('Failed to deliver packet: <TXStatus.NWK_ROUTE_DISCOVERY_FAILED: 208>'), DeliveryError('Failed to deliver packet: <TXStatus.NWK_ROUTE_DISCOVERY_FAILED: 208>')]
2022-10-19 08:05:32.057 WARNING (MainThread) [homeassistant.components.zha.core.channels.base] [0x67BB:1:0x0006]: async_initialize: all attempts have failed: [DeliveryError('Failed to deliver packet: <TXStatus.NWK_ROUTE_DISCOVERY_FAILED: 208>'), DeliveryError('Failed to deliver packet: <TXStatus.NWK_ROUTE_DISCOVERY_FAILED: 208>'), DeliveryError('Failed to deliver packet: <TXStatus.NWK_ROUTE_DISCOVERY_FAILED: 208>'), DeliveryError('Failed to deliver packet: <TXStatus.NWK_ROUTE_DISCOVERY_FAILED: 208>')]
2022-10-19 08:05:32.069 WARNING (MainThread) [homeassistant.components.zha.core.channels.base] [0x0E1E:1:0x0300]: async_initialize: all attempts have failed: [DeliveryError('Failed to deliver packet: <TXStatus.NWK_ROUTE_DISCOVERY_FAILED: 208>'), DeliveryError('Failed to deliver packet: <TXStatus.NWK_ROUTE_DISCOVERY_FAILED: 208>'), DeliveryError('Failed to deliver packet: <TXStatus.NWK_ROUTE_DISCOVERY_FAILED: 208>'), DeliveryError('Failed to deliver packet: <TXStatus.NWK_ROUTE_DISCOVERY_FAILED: 208>')]

and

2022-10-19 08:38:12.711 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [281472475170128] Error handling message: Unknown error (unknown_error)
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/components/websocket_api/decorators.py", line 27, in _handle_async_response
await func(hass, connection, msg)
File "/usr/src/homeassistant/homeassistant/components/zha/api.py", line 1065, in websocket_update_zha_configuration
if not data_to_save[CUSTOM_CONFIGURATION][section]:
KeyError: 'zha_alarm_options'

I can see from a previous backup that no ID's or values have changed. zigpy-deconz version has updated this month, but it seems to have been running fine since the 13th Oct - 19th Oct.

Copy of my backup:

{
    "backup_time": "2022-10-19T07:42:29.784030+00:00",
    "network_info": {
        "extended_pan_id": "00:21:2e:ff:ff:07:74:f3",
        "pan_id": "46D3",
        "nwk_update_id": 0,
        "nwk_manager_id": "0000",
        "channel": 15,
        "channel_mask": [
            15
        ],
        "security_level": 5,
        "network_key": {
            "key": "af:ab:fe:0d:6b:5b:2a:6d:37:81:66:2d:53:7a:12:72",
            "tx_counter": 0,
            "rx_counter": 0,
            "seq": 0,
            "partner_ieee": "ff:ff:ff:ff:ff:ff:ff:ff"
        },
        "tc_link_key": {
            "key": "5a:69:67:42:65:65:41:6c:6c:69:61:6e:63:65:30:39",
            "tx_counter": 0,
            "rx_counter": 0,
            "seq": 0,
            "partner_ieee": "00:21:2e:ff:ff:07:74:f3"
        },
        "key_table": [],
        "children": [],
        "nwk_addresses": {},
        "stack_specific": {},
        "metadata": {
            "deconz": {
                "version": 643303168
            }
        },
        "source": "zigpy-deconz@0.19.0"
    },
    "node_info": {
        "nwk": "0000",
        "ieee": "00:21:2e:ff:ff:07:74:f3",
        "logical_type": "coordinator"
    }
}

Answer 1 · 2022-10-19T14:18:16.000Z

I'm considering reinstalling ZHA entirely, and just setting everything up again.
Although I'd still love to know if I can find out what caused last night's issues, so that I can avoid or prepare in case it happens again. I'm unfortunately stumped, but should be able to help debug further with some help if anyone's available to.
Cheers!

Answer 2 · 2022-10-19T17:16:38.000Z

Perhaps relevant, my symptoms seem to mirror #169
Although I have definitely not attempted migrating or switching hardware ever.

Answer 3 · 2022-10-19T17:17:20.000Z

Can you upload your /config/zigbee.db file? I'm curious to see if your Conbee perhaps changed some critical network setting on its own.

Answer 4 · 2022-10-19T17:20:48.000Z

I would also recommend upgrading your Conbee's firmware.

Answer 5 · 2022-10-19T18:13:50.000Z

Thanks @puddly, here's my current zigbee.db (Oct 19th) with the majority of lights and a few switches temporarily fixed: https://drive.google.com/file/d/1GNh6GTsEaqa97-NYBq_gCNXaDn2uIFGh/view?usp=sharing

~~I have gdrive HA backups running, would a previous version of zigbee.db be helpful to compare against?~~
Here's a backup of my zigbee.db from two days ago (Oct 17th) before I had the issues, perhaps it'll help to compare with: https://drive.google.com/file/d/1Y6qrKDa6JnwCYHjJ2RmDvjwIpeViKvM1/view?usp=sharing

Would you have any recommended ways of updating Conbee firmware? I'm running things on a HA blue if that matters.
I'm guessing this guide is most relevant, I'll try that now: https://github.com/dresden-elektronik/deconz-rest-plugin/wiki/Update-deCONZ-manually
Done ✅

GCFFlasher V3_17 (c) dresden elektronik ingenieurtechnik gmbh
Reboot device COM3 (ConBee II)
deCONZ firmware version 26580700
R21B18 Bootloader
Vers: 2.07
build: Jun 17 2019
flashing 163230 bytes: |==============================|
verify: .
SUCCESS
Wait 10 seconds until application starts
deCONZ firmware version 26780700

Answer 6 · 2022-10-19T18:17:22.000Z

@puddly I don't mean to confuse things here, but perhaps this issue I raised against frontend for ZHA config is also related. home-assistant/frontend#14106

I'm assuming not, but I'm also stumped on that one, hence why I'm considering a full re-install of ZHA.

Answer 7 · 2022-10-19T18:57:36.000Z

I'm unfortunately seeing no difference between the two, only a single network configuration has ever been seen by zigpy. After upgrading firmware, have you tried taking and restoring a ZHA backup?

Answer 8 · 2022-10-19T19:07:07.000Z

Thanks for checking @puddly. I've never restored from a ZHA backup before, but it seems like I have everything in gdrive backups. I wonder have I messed my chances up though, by re-pairing a lot of devices today?

Bit of an issue now after upgrading conbee: HA is no longer seeing it in hardware 🤔 Even after a reboot.
Perhaps there's no support for the latest firmware deCONZ_ConBeeII_0x26780700.bin.GCF.

Answer 9 · 2022-10-19T19:10:19.000Z

Restoring from a ZHA backup would reset the radio and re-write the network settings.

The latest firmware is supported. Is your Conbee on a USB 2.0 extension cable? It sometimes gets into a state where it bootloops.

Answer 10 · 2022-10-19T19:30:12.000Z

Restoring from a ZHA backup would reset the radio and re-write the network settings.

The latest firmware is supported. Is your Conbee on a USB 2.0 extension cable? It sometimes gets into a state where it bootloops.

It's never been on an extension cable no, I've never seemed to need it with my HA Blue, but I have just ordered one for good measure.
I've tried a full system reboot, and plugged the conbee in and out, but still can't see it in hardware anywhere. And ZHA is obviously failing now to connect.

I'm literally sitting in darkness with no plan B 😅

I appreciate your help!

Logger: homeassistant.components.zha.core.gateway
Source: components/zha/core/gateway.py:172
Integration: Zigbee Home Automation ([documentation](https://www.home-assistant.io/integrations/zha), [issues](https://github.com/home-assistant/home-assistant/issues?q=is%3Aissue+is%3Aopen+label%3A%22integration%3A+zha%22))
First occurred: 20:24:42 (3 occurrences)
Last logged: 20:25:01

Couldn't start deCONZ = dresden elektronik deCONZ protocol: ConBee I/II, RaspBee I/II coordinator (attempt 1 of 3)
Couldn't start deCONZ = dresden elektronik deCONZ protocol: ConBee I/II, RaspBee I/II coordinator (attempt 2 of 3)
Couldn't start deCONZ = dresden elektronik deCONZ protocol: ConBee I/II, RaspBee I/II coordinator (attempt 3 of 3)
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/serial/serialposix.py", line 322, in open
    self.fd = os.open(self.portstr, os.O_RDWR | os.O_NOCTTY | os.O_NONBLOCK)
FileNotFoundError: [Errno 2] No such file or directory: '/dev/serial/by-id/usb-dresden_elektronik_ingenieurtechnik_GmbH_ConBee_II_DE2447556-if00'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/zha/core/gateway.py", line 172, in async_initialize
    self.application_controller = await app_controller_cls.new(

Answer 11 · 2022-10-19T19:36:24.000Z

Is it still visible from within Windows? Perhaps you can downgrade to the previous version.

Answer 12 · 2022-10-19T19:44:08.000Z

It is still seen in windows, yes!
I'll try with deCONZ_ConBeeII_0x26720700.bin.GCF perhaps? (2021 version)

GCFFlasher V3_17 (c) dresden elektronik ingenieurtechnik gmbh
Path             | Vendor | Product | Serial     | Type
-----------------+--------+---------+------------+-------
\\.\COM3         | 0x1CF1 | 0x0030  | DE2447556  | ConBee II

Answer 13 · 2022-10-19T19:49:33.000Z

Alright, flashed. ✅

Can I check with you: What steps would you take yourself when reconnecting the conbee stick to your HA?
Should I connect it via usb, reboot system (via hardware settings), wait for reboot, and then check the hardware list?
That's what I've been attempting, with a search for 'conbee', 'deconz', 'conz', 'zigbee', 'zha', which results in nothing, and usb1 and 2 seem to be empty too.

GCFFlasher_V3_17> .\GCFFlasher -d COM3 -t 60 -f deCONZ_ConBeeII_0x26720700.bin.GCF
GCFFlasher V3_17 (c) dresden elektronik ingenieurtechnik gmbh
Reboot device COM3 (ConBee II)
deCONZ firmware version 26780700
R21B18 Bootloader
Vers: 2.07
build: Jun 17 2019
flashing 164227 bytes: |==============================|
verify: .
SUCCESS
Wait 10 seconds until application starts
deCONZ firmware version 26720700

Answer 14 · 2022-10-19T19:50:54.000Z

You generally shouldn't need to reboot anything. Do other USB devices get recognized by your Blue?

Answer 15 · 2022-10-19T19:51:25.000Z

Ah nevermind!, I see it in hardware again now after installing that 2021 version of deconz! Thanks for suggesting 🙌

/dev/serial/by-id/usb-dresden_elektronik_ingenieurtechnik_GmbH_ConBee_II_DE2447556-if00

Answer 16 · 2022-10-19T19:56:29.000Z

And after a reload of ZHA integration, I can control my re-paired lights from today 👏

However, I'm still not connecting to all my other devices that were working before last night's issue unfortunately.
And I'm also still unable to update my config (home-assistant/frontend#14106)

What would you try next? I'm thinking a complete uninstall of ZHA, and just go for a full re-setup for all my devices manually. 🤔

Answer 17 · 2022-10-19T19:57:55.000Z

Try downloading a ZHA backup and then "migrating" by restoring it. If that doesn't work, there's not much else I can think of, unfortunately.

Answer 18 · 2022-10-19T20:01:38.000Z

I just ran a migration using a backup file from the 13th. I'm still only able to control the re-paired devices from today, and the rest are unavailable.

Should I have tried a different migration method there?

Answer 19 · 2022-10-19T20:02:30.000Z

Have you tried just rebooting the affected devices, without factory resetting them?

Answer 20 · 2022-10-19T20:06:53.000Z

No, and to be honest I don't even know what you mean by rebooting a device?
I can't see reboot option in device settings, or do you mean physically cycling power like a bulb? (If so, yes I have tried physical power cycling on devices on mains. For remotes/motion, all I know is repairing mode + adding via zha again)

Answer 21 · 2022-10-19T21:13:14.000Z

So after deleting, and re-adding the integration, it wiped my device names, and I could still only communicate with the re-paired devices, and continued to have that separate configuration bug.

So I deleted and re-added again, but this time starting with as a new network. Straight away, I'm able to update the configuration for ZHA, and I also see a config section for alarm panels that I've never seen before.

Again, I'm not sure if this is related to last night's issues, but it seems to me that I've been stuck in some broken ZHA migration perhaps 🤔.

Answer 22 · 2022-10-19T21:56:51.000Z

Truthfully, this isn't unheard of for the Conbee. I'd get a different Zigbee coordinator (like the CC2652) and utilize the migration feature.

Answer 23 · 2022-10-20T00:00:04.000Z

Thanks @puddly , really appreciate you trying!
Are you running a particular CC2652 yourself? I was eyeing the Sonoff's Zigbee 3 in the aluminium case, but then came across reports of issues with ikea devices, which I'd have a lot of.

Answer 24 · 2022-10-20T00:13:16.000Z

I'm not aware of any issues with IKEA devices that are specific to the CC2652. Which ones are you referring to?

Answer 25 · 2022-10-20T00:30:07.000Z

The Sonoff Zigbee 3, I was close to ordering it this morning, but came across a few reports of issues with Ikea: https://community.home-assistant.io/t/sonoff-zigbee-usb-dongle-not-working-with-ikea-tradfri-buttons/385878

Mind if I ask what you're running yourself?

Answer 26 · 2022-10-20T00:33:13.000Z

I'm not aware of any firmware issue that would cause this so it's likely just a tricky device. In any case, if it works now it will work the same way after the migration.

Mind if I ask what you're running yourself?

I work on the radio libraries so I rotate between the LAUNCHXL-CC1352P2 (basically the same thing as the CC2652), the Conbee II, and the SkyConnect (EFR32).