Upgrading to MyNode 0.3.22 broke some apps (RTL, LightningTerminal, Pool, Loop) and downgrading to LND 0.16.4 failed
tukac opened this issue · 7 comments
Upgrading MyNode to 0.3.22 (to LND 0.17.0-beta) broke some apps (RTL, LightningTerminal, Pool, Loop).
Device: amd64, 2Tb, 32Gb
RTL error log:
-- Logs begin at Sat 2023-11-04 06:58:27 CDT, end at Sat 2023-11-04 07:23:23 CDT. --
Nov 04 07:23:22 myNode systemd[1]: Failed to start RTL.
Nov 04 07:23:22 myNode systemd[1]: rtl.service: Failed with result 'timeout'.
Nov 04 07:23:19 myNode rtl[30009]: + sleep 3s
Nov 04 07:23:19 myNode rtl[30009]: + sync
Nov 04 07:23:19 myNode rtl[30009]: + sed -i s/testnet/mainnet/g /mnt/hdd/mynode/rtl/RTL-Config.json
Nov 04 07:23:19 myNode rtl[30009]: + '[' -f /mnt/hdd/mynode/settings/.testnet_enabled ']'
Nov 04 07:23:19 myNode rtl[30009]: + sed -i 's/"multiPassHashed":.*/"multiPassHashed": "c59ceca0a7f5022bde5a0e1697d2042ef00cc335c3d0ac6ffc91b1e7ccc5d567",/g' /mnt/hdd/mynode/rtl/RTL-Config.json
Nov 04 07:23:19 myNode rtl[30009]: + HASH=c59ceca0a7f5022bde5a0e1697d2042ef00cc335c3d0ac6ffc91b1e7ccc5d567
Nov 04 07:23:19 myNode rtl[30009]: ++ cat /home/bitcoin/.mynode/.hashedpw
Nov 04 07:23:19 myNode rtl[30009]: + '[' -f /home/bitcoin/.mynode/.hashedpw ']'
Nov 04 07:23:19 myNode rtl[30009]: + '[' '!' -f /mnt/hdd/mynode/rtl/update_settings_1 ']'
Nov 04 07:23:19 myNode rtl[30009]: + RTL_CONFIG_UPDATE_NUM=1
Nov 04 07:23:19 myNode rtl[30009]: + '[' '!' -f /mnt/hdd/mynode/rtl/RTL-Config.json ']'
Nov 04 07:23:19 myNode rtl[30009]: + '[' '!' -L /opt/mynode/RTL/RTL-Config.json ']'
Nov 04 07:23:19 myNode rtl[30009]: + chown -R bitcoin:bitcoin /mnt/hdd/mynode/rtl_backup
Nov 04 07:23:19 myNode rtl[30009]: + chown -R bitcoin:bitcoin /mnt/hdd/mynode/rtl
Nov 04 07:23:19 myNode sudo[30030]: pam_unix(sudo:session): session closed for user bitcoin
Nov 04 07:23:19 myNode sudo[30030]: pam_unix(sudo:session): session opened for user bitcoin by (uid=0)
Nov 04 07:23:19 myNode sudo[30030]: root : TTY=unknown ; PWD=/opt/mynode/RTL ; USER=bitcoin ; COMMAND=/usr/bin/mkdir -p /mnt/hdd/mynode/rtl
Nov 04 07:23:19 myNode sudo[30030]: PAM adding faulty module: pam_chksshpwd.so
Nov 04 07:23:19 myNode sudo[30030]: PAM unable to dlopen(pam_chksshpwd.so): /lib/security/pam_chksshpwd.so: cannot open shared object file: No such file or directory
Nov 04 07:23:19 myNode rtl[30009]: + sudo -u bitcoin mkdir -p /mnt/hdd/mynode/rtl
Nov 04 07:23:19 myNode sudo[30028]: pam_unix(sudo:session): session closed for user bitcoin
Nov 04 07:23:19 myNode sudo[30028]: pam_unix(sudo:session): session opened for user bitcoin by (uid=0)
Nov 04 07:23:19 myNode sudo[30028]: root : TTY=unknown ; PWD=/opt/mynode/RTL ; USER=bitcoin ; COMMAND=/usr/bin/mkdir -p /opt/mynode/RTL
Nov 04 07:23:19 myNode sudo[30028]: PAM adding faulty module: pam_chksshpwd.so
Nov 04 07:23:19 myNode sudo[30028]: PAM unable to dlopen(pam_chksshpwd.so): /lib/security/pam_chksshpwd.so: cannot open shared object file: No such file or directory
Nov 04 07:23:19 myNode rtl[30009]: + sudo -u bitcoin mkdir -p /opt/mynode/RTL
Nov 04 07:23:19 myNode rtl[30009]: + set -e
Nov 04 07:23:19 myNode rtl[30009]: Linux myNode 5.10.0-0.deb10.16-amd64 #1 SMP Debian 5.10.127-2~bpo10+1 (2022-07-28) x86_64 GNU/Linux
Nov 04 07:23:19 myNode rtl[30009]: /usr/share/mynode/mynode_device_info.sh: line 17: /proc/device-tree/model: No such file or directory
Nov 04 07:23:18 myNode systemd[1]: rtl.service: Control process exited, code=killed, status=15/TERM
Nov 04 07:23:18 myNode systemd[1]: rtl.service: Start-pre operation timed out. Terminating.
Nov 04 07:23:18 myNode rtl[28220]: LND not synced, sleeping for 60 seconds...
Nov 04 07:22:18 myNode rtl[28220]: LND not synced, sleeping for 60 seconds...
Nov 04 07:21:18 myNode rtl[28220]: LND not synced, sleeping for 60 seconds...
Nov 04 07:21:18 myNode rtl[28220]: Checking if LND is synced...
Nov 04 07:21:18 myNode rtl[28220]: Linux myNode 5.10.0-0.deb10.16-amd64 #1 SMP Debian 5.10.127-2~bpo10+1 (2022-07-28) x86_64 GNU/Linux
Nov 04 07:21:18 myNode rtl[28220]: /usr/share/mynode/mynode_device_info.sh: line 17: /proc/device-tree/model: No such file or directory
Nov 04 07:21:18 myNode rtl[28219]: Not shutting down!
Nov 04 07:21:18 myNode rtl[28219]: Checking if device is shutting down...
Nov 04 07:21:18 myNode systemd[1]: Starting RTL...
Nov 04 07:21:18 myNode systemd[1]: Stopped RTL.
Nov 04 07:21:18 myNode systemd[1]: rtl.service: Scheduled restart job, restart counter is at 7.
Tried to downgrade to LND 0.16.4.
Several different attempts to downgrade all failed. Customising LND version didn't succeed when done from:
- column with all apps listed and Customize button, version v0.16.4 was pinned but after Upgrade nothing changed
- text box on bottom of the page, entered
LND_VERSION="v0.16.4", after reboot app menu shows 0.16.4 as latest and none LND installed although LND 0.17.0-beta was running fine - from Manage Applications page neither Restart, Upgrade nor Reinstall actually installed 0.16.4 although now 0.16.4 was indicated as latest version and still
"not installed"status remained for LND although LND 0.17.0-beta was running fine.
This is from latest Upgrade attempt:
++ LND_UPGRADE_MANIFEST_GUGGERO_SIG_URL=https://github.com/lightningnetwork/lnd/releases/download/v0.16.4/manifest-guggero-v0.16.4.sig
++ LND_UPGRADE_MANIFEST_ROASBEEF_SIG_URL=https://github.com/lightningnetwork/lnd/releases/download/v0.16.4/manifest-roasbeef-v0.16.4.sig
++ LND_UPGRADE_MANIFEST_URL=https://github.com/lightningnetwork/lnd/releases/download/v0.16.4/manifest-v0.16.4.txt
++ LND_LATEST_VERSION_FILE=/home/bitcoin/.mynode/lnd_version_latest
++ LND_VERSION_FILE=/home/bitcoin/.mynode/lnd_version
++ LND_VERSION=v0.16.4
+++ echo v0.16.4
+++ version=v0.16.4
++++ cat /mnt/hdd/mynode/settings/lnd_version_latest_custom
+++ '[' -f /mnt/hdd/mynode/settings/lnd_version_latest_custom ']'
+++ version=v0.16.4
++++ cat /home/bitcoin/.mynode/lnd_version_latest_custom
+++ '[' -f /home/bitcoin/.mynode/lnd_version_latest_custom ']'
+++ local version=v0.17.0-beta
+++ local short_name=lnd
+++ local official_version=v0.17.0-beta
+++ get_app_version v0.17.0-beta lnd
++ LND_VERSION=v0.17.0-beta
Still stuck on 0.17.0-beta, lncli getinfo still shows "version": "0.17.0-beta commit=v0.17.0-beta", every option tried to downgrade has failed.
RTL, LightningTerminal, Pool and Loop not working.
The 'wait_on_lnd.sh' script seems to be missing a whitespace in the grep pattern. Changing
grep 'synced_to_chain": true'
to
grep 'synced_to_chain": true'
solved it for me
Spot on.
Running sudo find / -type f -name 'wait_on_lnd.sh' found two instances:
/usr/bin/wait_on_lnd.sh
/opt/upgrade/out/rootfs_debian/usr/bin/wait_on_lnd.sh
I modified first one: sudo nano /usr/bin/wait_on_lnd.sh
from
grep 'synced_to_chain": true'
to
grep 'synced_to_chain": true'
This solved issue with RTL, LightningTerminal, Pool and Loop.
Manual downgrade issue remains.
There was database migration in lnd 0.17.0. These are not backwards compatible. Downgrade to 0.16.4 would not work eitherway.
The output I see from "lncli getinfo" only has one space. Can you share your output? I can make the regex check for multiple spaces.
As @babikpatient6 already mentioned, LND downgrades are not always possible due to database conversions.
It seems all my lncli getinfo outputs have two spaces, here is excerpt:
"best_header_timestamp": "1699219991",
"synced_to_chain": true,
"synced_to_graph": true,
"testnet": false,
"chains": [
{
"chain": "bitcoin",
"network": "mainnet"
}
],
Weird. I found it on one node but not all of them. I just released an update with a fix. Thanks for letting me know.
Thank you for quick patch, it solved the issue with lnd related apps.
I am closing this ticket.