[ANNOUNCEMENT] Zigbee-herdsman: the next zigbee-shepherd
Koenkk opened this issue · 49 comments
As some of you have already noticed, lately I've been putting lots of efforts in refactoring zigbee-shepherd. Zigbee-shepherd is the Javascript Zigbee library were zigbee2mqtt is based up-on.
My biggest reason to do this is that, in my opinion, zigbee-shepherd was not maintainable and extensible. This made it very hard to fix bugs and add new features, which made it hard to take zigbee2mqtt to the next level.
I ended up in completely rewriting zigbee-shepherd from scratch. Because of that I also gave it a different name, zigbee-herdsman. Just to give an idea of the effort that I've put into it, this job took me 100+ hours spread over 2 months.
Zigbee-herdsman will land into the zigbee2mqtt (dev branch) and zigbee-shepherd-converters after zigbee2mqtt 1.6. After this zigbee-shepherd-converters will be renamed to zigbee-herdsman-converters, as zigbee-shepherd-converters and zigbee-herdsman are not compatible.
What will I as a user notice from it?
- Improved stability and predictability
- Improved Xiaomi devices pairing (should be similar to the official Xiaomi gateway now, even for Mija devices, bye bye
Cannot get node descriptor for...error 😺) - Improved logging, especially during pairing devices (earlier shows when a device has been connected and we can now show if a device has been paired successfully)
- Improved error handling (e.g. unplugging the stick will stop zigbee2mqtt now)
- (future) Possibility to support other adapters than TI Z-stack adapters
- (future) Keep track of device bindings
What will I as a developer notice from it?
- Written in Typescript; makes it easier to detect errors earlier in the development process -> less bugs
- 100% test coverage; makes it easier to detect errors earlier in the development process -> less bugs
- Reduced amount of dependencies (and especially not maintained dependencies)
- Simplified Zigbee API -> adding devices to zigbee-shepherd-converters will be easier
- Automatic iAS zone enrolment -> no need to do it in zigbee-shepherd-converters anymore -> easier to add support for iAS devices
- Push down code from zigbee2mqtt into zigbee-herdsman (= less zigbee2mqtt code size and other programs based on zigbee-herdsman are able to benefit from it).
- Drastically reduced and simplified zigbee-shepherd codebase:
- From 17231 to 11273 lines (= 35%), but as a big part of the code are long definition files (example), it would be fair to count without these files, resulting in: From 7675 to 3739 lines (=52% code reduction)
Nice!
Looking forward to trying 1.6.
Nice!
Looking forward to trying 1.6.
Just to be clear, it will NOT be included in 1.6, but will be included in the dev branch after the 1.6 release. (we first need some time to test before it can be released).
Great job!
Thank you a million times for all the hard work and thank you to everybody involved in this great project.
That sounds amazing!!
Thank you so much for such a great project and work!
What will be the optimal hardware (CC2530, ...) for the new version?
Koen, Thanks for your great work. I still have a love-hate relationship with zigbee. I've quite a large network, about 37 devices, still growing. Stability is increasing because of your hard work. Less devices are getting lost. My network is expanding, 3 floors and a garden. Which never could have happened without zigbee2mqtt! Thanks thanks thanks!
Improved Xiaomi devices pairing (should be similar to the official Xiaomi gateway now, even for Mija devices, bye bye Cannot get node descriptor for... error 😺
Can't wait to test this... I have more than 50 Devices, lots of them Xiaomi/Aqara. 👍 Herd is still growing and pairing Xiaomi has become a pain in the #$§!!
Maybe, a hint to Donation is useful. Come on Guys: Just think about the difference sending for any given device (f.e. hue vs tradfri)...
Maybe, a hint to Donation is useful. Come on Guys: Just think about the difference sending for any given device (f.e. hue vs tradfri)...
That was on my list. Thanks for reminding.
zigbee-herdsman has now landed in the dev branch, let the testing begin :)
Hi,
Is there any breaking change? I udated dev branch and now cannot start:
Sep 09 20:21:41 SmartHome systemd[1]: Started zigbee2mqtt.
Sep 09 20:21:42 SmartHome npm[7178]: > zigbee2mqtt@1.6.0 start /opt/zigbee2mqtt
Sep 09 20:21:42 SmartHome npm[7178]: > node index.js
Sep 09 20:21:43 SmartHome npm[7178]: zigbee2mqtt:info 9/9/2019, 8:21:43 PM Logging to directory: '/opt/zigbee2mqtt/data/log/2019-09-09.20-21-42'
Sep 09 20:21:43 SmartHome npm[7178]: zigbee2mqtt:info 9/9/2019, 8:21:43 PM Starting zigbee2mqtt version 1.6.0 (commit #d83085e)
Sep 09 20:21:43 SmartHome npm[7178]: zigbee2mqtt:info 9/9/2019, 8:21:43 PM Starting zigbee-herdsman...
Sep 09 20:21:44 SmartHome npm[7178]: zigbee2mqtt:error 9/9/2019, 8:21:44 PM Error while starting zigbee-herdsman
Sep 09 20:21:44 SmartHome npm[7178]: zigbee2mqtt:error 9/9/2019, 8:21:44 PM Error: Backup is only supported for Z-Stack 3
Sep 09 20:21:44 SmartHome npm[7178]: at Object. (/opt/zigbee2mqtt/node_modules/zigbee-herdsman/dist/adapter/z-stack/adapter/backup.js:78:19)
Sep 09 20:21:44 SmartHome npm[7178]: at Generator.next ()
Sep 09 20:21:44 SmartHome npm[7178]: at fulfilled (/opt/zigbee2mqtt/node_modules/zigbee-herdsman/dist/adapter/z-stack/adapter/backup.js:4:58)
Sep 09 20:21:44 SmartHome npm[7178]: zigbee2mqtt:error 9/9/2019, 8:21:44 PM Failed to start zigbee
Sep 09 20:21:44 SmartHome npm[7178]: zigbee2mqtt:error 9/9/2019, 8:21:44 PM Exiting...
Sep 09 20:21:44 SmartHome npm[7178]: npm ERR! code ELIFECYCLE
Sep 09 20:21:44 SmartHome npm[7178]: npm ERR! errno 1
Sep 09 20:21:44 SmartHome npm[7178]: npm ERR! zigbee2mqtt@1.6.0 start: node index.js
Sep 09 20:21:44 SmartHome npm[7178]: npm ERR! Exit status 1
Sep 09 20:21:44 SmartHome npm[7178]: npm ERR!
Sep 09 20:21:44 SmartHome npm[7178]: npm ERR! Failed at the zigbee2mqtt@1.6.0 start script.
Sep 09 20:21:44 SmartHome npm[7178]: npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
Sep 09 20:21:44 SmartHome npm[7178]: npm ERR! A complete log of this run can be found in:
Sep 09 20:21:44 SmartHome npm[7178]: npm ERR! /home/pi/.npm/_logs/2019-09-09T18_21_44_648Z-debug.log
Sep 09 20:21:44 SmartHome systemd[1]: zigbee2mqtt.service: Main process exited, code=exited, status=1/FAILURE
Sep 09 20:21:44 SmartHome systemd[1]: zigbee2mqtt.service: Failed with result 'exit-code'.
Sep 09 20:21:44 SmartHome systemd[1]: zigbee2mqtt.service: Service RestartSec=100ms expired, scheduling restart.
Sep 09 20:21:44 SmartHome systemd[1]: zigbee2mqtt.service: Scheduled restart job, restart counter is at 2.
Can you provide the log when running with DEBUG=* npm start? (please post it on pastebin)
I have some issues with device locking. I'm not sure if it's my setup or not. I have to physically disconnect my USB device and reconnect it before I can restart zigbee2mqtt.
And it looks like I've just pastebinned my network key...
Can you provide the log when running with
DEBUG=* npm start? (please post it on pastebin)
Sure, here it is: https://pastebin.com/CUF7MBMZ
@Koenkk Next time it happens I will post the data.
Another issue with an unhandled promise rejection: https://pastebin.com/S6bj9VY5
@santirguez issue should be fixed in the latest dev branch (make sure to do a rm -rf node_modules && npm install after updating).
@BabyDino I see that you are also at least suffering from the same issue as @santirguez, (Error: Backup is only supported for Z-Stack 3). Should be fixed in latest dev, can you check if it also fixed the other issue. Regarding the resource busy, it probably happens because zigbee2mqtt fails to start (because of the issue) and therefore keeping the port locked. So that should fix itself automatically when we can get zigbee2mqtt to start :)
9/9/2019, 7:38:57 PM - debug: Loaded state from file /app/data/state.json
9/9/2019, 7:38:57 PM - debug: Saving state to file /app/data/state.json
9/9/2019, 7:38:57 PM - info: Starting zigbee2mqtt version 1.6.0 (commit #d83085e)
9/9/2019, 7:38:57 PM - info: Starting zigbee-herdsman...
9/9/2019, 7:38:57 PM - debug: Using zigbee-herdsman with settings: '{"network":{"panID":6754,"extenedPanID":[221,221,221,221,221,221,221,221],"channelList":[11],"networkKey":"HIDDEN"},"databasePath":"/app/data/database.db","backupPath":"/app/data/coordinator_backup.json","serialPort":{"baudRate":115200,"rtscts":true,"path":"/dev/ttyACM0"}}'
9/9/2019, 7:38:59 PM - error: Error while starting zigbee-herdsman
Cannot get more from debug log. Running latest docker container on RPi 3.
@way2many probably same issue as @santirguez, please update to the latest dev (you should see commit #411f1fa). If you are on docker, wait until https://travis-ci.org/Koenkk/zigbee2mqtt/builds/582844293?utm_source=github_status&utm_medium=notification has been finished. (max 30 minutes from now).
EDIT: docker image is ready now.
@Koenkk lsof /dev/ttyUSB2:
node 17737 pi 22uW CHR 188,2 0t0 17006905 /dev/ttyUSB2/edit: after killing the process the error remains.
@BabyDino I think you also need to unplug the usb, please see my explanation from #1888 (comment)
Updated to latest docker container and everything is working fine. Thanks @Koenkk
Startup was blazing fast!
@Koenkk Makes sense. It's running now. I have a lot of Xiaomi stuff and still some pairings to do. Do you want individual issues created in the zigbee-hersman repo or report in this issue for now?
@Koenkk I'm not sure what you have done, but this is lightning fast!! I have 2 spots in the hallway and they are turning on simultaneously without delay (instead of one before the other). Everything reacts real-time to commands. Will test stability in the coming days. A IKEA Tradfri lamp just paired flawlessly (I couldn't get it paired on -shepherd).
Thank you for your hard work!
Thanks @Koenkk! First impression is great. It's fast, some devices were impossible to pair and now it's done in an eye wink. Thanks!
Great work @Koenkk
After update I had to unplug the stick to get zigbee-herdsman started, but then the sytem was up for normal operationing very fast. It is very stable, even without a router (which was necessary before)
Thanks!
Latest-dev is working GREAT for me here, and I'm awe-struck by the speed. Well done! I'm struggling with joining some devices (Cree Connected bulbs, which worked before), and if I can't get that worked out I'll file a separate issue. I have some debug log spelunking to finish before I do.
@santirguez issue should be fixed in the latest dev branch (make sure to do a
rm -rf node_modules && npm installafter updating).
Hi @Koenkk. It was working perfectly! Now after a couple of day I lost around 4 Xiaomi movement sensors (not reporting state in zigbee2mqtt)
I am trying to remove them to repair them again, but every time I try to remove a sensro I get this error:
PM Failed to remove "XXXXX" (Error: AREQ - ZDO - mgmtLeaveRsp after 10000ms)
Is there any way to remove them without loose everything else?
Thank you for your work!!
Best regards,
This makes sense as the device is sleeping its not responding to requests. If you hold the reset button on the device it will remove itself. After that you can repair.
Latest-dev is working GREAT for me here, and I'm awe-struck by the speed. Well done! I'm struggling with joining some devices (Cree Connected bulbs, which worked before), and if I can't get that worked out I'll file a separate issue. I have some debug log spelunking to finish before I do.
@wixoff @Koenkk I can confirm this behavior with CREE Connected bulbs. So long as they are already joined they work fine. I had a couple to add to the network, here's what I did:
- revert to
latestdocker image - join new CREE bulbs like normal
- change to
latest-devwith existingdatabase.db - new CREE bulbs work as expected
@r1cht3r same here with GledOpto bulbs. Pairing works fine in 1.5, moving database to 1.6 gives me a full network for a while but loosing contact with some of the bulbs after a few minutes and no way to get it paired again. Last night I ran with 1.5, changing color each 5 minutes and all Gledoptos are still alive.
I changed my scripts to prevent bursting, because this is definately causing problems. (max 1 publish per second now).
@Koenkk which data do you need to see what's wrong, I can same db-files from different stages. Running 1.5, after starting 1.6, after lost contact on 1.6, all logs files?
@bochelork Can you provide the log when running with DEBUG=* npm start when pairing the device? Please provide the log on pastebin.com and link it here.
@Koenkk With a working 1.5 database I started with latest dev. After starting I switched off 7 devices with zibee2mqtt with 0.5 seconds interval.
Here I switch off the 230v power of 6 devices, I switch on after 10 seconds. This should announce the devices.
Switching off by zigbee2mqtt 7 devices again. Only 2 respond.
Reset the bulbs. 5 times 230v power on/off
Again switch off by zigbeemqtt, only 1 responds.
After this I switched back to 1.5, 5 out of 7 device respond normally. Have to repair the others....
@bochelork can you try with: https://github.com/Koenkk/Z-Stack-firmware/tree/master/coordinator/Z-Stack_Home_1.2/bin/source_routing? Before executing your tests make sure to turn each device on/off via zigbee2mqtt once. This ensures that the coordinator knows the routes.
hi there,
giving a try to herdsman/dev, looking very good so far, where are the devices stored ? in states.json ?
because I've restarted the daemon and I had to re-pair everything again.
edit: I think they're stored in states.json, in my z2m start script I used to rm states.json because of #2033. Could you confirm this ? :-)
Only states are stored there, devices are stored in database.db
I switched over to try the dev version with "herdsman" but somehow I lost a few devices, they should be connected but don't give any signal and repairing doesn't seem to work.
Any thoughts how I can help to give you more additional info to debug this?
The devices what I'm trying to repair are Ikea E1746 Signal Repeater, Xiaomi WSDCGQ11LM and a Xiaomi MCCGQ01LM
Update: SOLVED
It seems pairing works best with full batteries :-)
The sensor was sending happily with the old battery but pairing was not possible.
I'm also having problems repairing Xiaomi Temperature Sensor (Xiaomi WSDCGQ11LM) :-(
I'm thinking of rebuilding my z2m setup (redoing Home Assistant and the works). How far along is the next release?
Is the dev any good for daily use?
(I have only Xiaomi sensors at the moment)
@erazor666 i have using it about 2-3 weeks and it is rock solid. Try it and you will not go back to the original shephard.
Yes, I have finally eliminated ALL dropped devices using a combination of the latest-dev (post-1.6.0) z2m with zigbee-herdsman and the Zigbee 3.0.x firmware on a CC2531.
As @L3H0 said, it's ROCK SOLID. And much faster!
Hi everyone,
I am having trouble when pairing a Sylvania device LIGHTIFY LED RGBW A19, immediately after pairing, it shows an Error "Cluster with key '64513' does not exist" and stops the zigbee2mqtt service.
I am new to this whole zigbee, mqtt, and linux enviroment... has anyone experimented with the same error?
@Emma-2409 should be fixed in the zigbee2mqtt dev branch (https://www.zigbee2mqtt.io/how_tos/how-to-switch-to-dev-branch.html)
@Emma-2409 should be fixed in the zigbee2mqtt dev branch (https://www.zigbee2mqtt.io/how_tos/how-to-switch-to-dev-branch.html)
Thanks Koenkk, it worked just fine, you have created something wonderful I must say ;)
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
anti-stale
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.