Unable to start Lightning Terminal Daemon due to Pool not starting
martinatime opened this issue · 8 comments
I have Lightning Terminal using a standalone LND and it had been working for a couple of months. Then sometime between 1-2 weeks ago something happened where the litd (running as a systemctl daemon) would fail to start. The issue seems to be related to a pool account. I attempted to start litd not using systemctl and it still fails to start due to the pool account issue.
Expected behavior
The expected behavior is that litd starts successfully with loop, pool, and faraday.
Actual behavior
litd fails to start
To reproduce
ran litd using the user that typically runs litd
System information
Part of Lightning Terminal
Not using Umbrel
Both UI and CLI when it is working
Version: 0.5.4-alpha commit=v0.5.4-alpha.0.20220114202858-525fe156d240
Downloaded the binary for Lightning Terminal for arm
Raspiberry Pi 4 / Raspbian 64 bullseye / SSD
Raspbian 64 11.2
I'm just curious if this is being looked at. I still have the same issue when I attempt to start litd. Is there any additional information that I can provide to help in investigating this?
2022-03-24 07:42:34.145 [ERR] LITD: Could not start subservers: unable to start account manager: unable to resume account <redacted>: unable to subscribe for account updates: checking pending batch failed: error removing pending batch artifacts: error abandoning channels from previous pending batch: error locating channel outpoint: no channel output found in batch tx for matched order <redacted>
I think this might be something that was fixed recently. So it shouldn't happen again on a newer version (I hope).
To enable you to start up again, can you try to recover the account? That will cancel any open orders but should allow you to start LiT again and continue using Pool: https://docs.lightning.engineering/lightning-network-tools/lightning-terminal/troubleshooting#pool-account-recovery
Thanks for the quick response. I'll give this a try this evening after work and I'll report back regardless of outcome.
ok, here's the problem. It seems that this process requires that the pool RPC be up and running but the whole problem is that while Lightning Terminal is starting, including pool, it fails and then shuts everything down. I attempted anyway with no success.
I then saw that there was a new version so I downloaded and installed that. It seemed to have more log messages but the same issue as the original log I attached.
2022-03-24 21:28:22.699 [INF] CLDB: Checking for schema update: latest_version=1, db_version=1
2022-03-24 21:28:22.699 [DBG] GRPC: [core] parsed scheme: ""
2022-03-24 21:28:22.699 [DBG] GRPC: [core] scheme "" not registered, fallback to default scheme
2022-03-24 21:28:22.699 [DBG] GRPC: [core] ccResolverWrapper: sending update to cc: {[{pool.lightning.finance:12010 <nil> 0 <nil>}] <nil> <nil>}
2022-03-24 21:28:22.700 [DBG] GRPC: [core] ClientConn switching balancer to "pick_first"
2022-03-24 21:28:22.700 [DBG] GRPC: [core] Channel switches to new LB policy "pick_first"
2022-03-24 21:28:22.700 [DBG] GRPC: [core] Subchannel Connectivity change to CONNECTING
2022-03-24 21:28:22.700 [INF] RPCS: Starting trader server
2022-03-24 21:28:22.700 [DBG] GRPC: [core] Subchannel picks a new address "pool.lightning.finance:12010" to connect
2022-03-24 21:28:22.701 [DBG] GRPC: [core] pickfirstBalancer: UpdateSubConnState: 0x48ae450, {CONNECTING <nil>}
2022-03-24 21:28:22.701 [DBG] GRPC: [core] Channel Connectivity change to CONNECTING
2022-03-24 21:28:22.774 [DBG] GRPC: [core] Subchannel Connectivity change to READY
2022-03-24 21:28:22.774 [DBG] GRPC: [core] pickfirstBalancer: UpdateSubConnState: 0x4d720b0, {READY <nil>}
2022-03-24 21:28:22.774 [DBG] GRPC: [core] Channel Connectivity change to READY
2022-03-24 21:28:22.946 [INF] RPCS: Connected to lnd node [REDACTED] with pubkey [REDACTED]
2022-03-24 21:28:22.958 [INF] ACCT: Watching spend of [REDACTED]:0 for account [REDACTED]
2022-03-24 21:28:23.256 [DBG] GRPC: [core] Subchannel Connectivity change to READY
2022-03-24 21:28:23.256 [DBG] GRPC: [core] pickfirstBalancer: UpdateSubConnState: 0x48ae450, {READY <nil>}
2022-03-24 21:28:23.257 [DBG] GRPC: [core] Channel Connectivity change to READY
2022-03-24 21:28:23.348 [DBG] AUCT: Connected successfully to server after 1 tries
2022-03-24 21:28:23.348 [INF] AUCT: Successfully connected to auction server
2022-03-24 21:28:23.457 [INF] AUCT: Shutting down auctioneer client
2022-03-24 21:28:23.457 [DBG] AUCT: Closing server stream
2022-03-24 21:28:23.457 [TRC] AUCT: Received msg=(*auctioneerrpc.ServerAuctionMessage)(nil), err=rpc error: code = Canceled desc = context canceled from server
2022-03-24 21:28:23.458 [DBG] GRPC: [core] Channel Connectivity change to SHUTDOWN
2022-03-24 21:28:23.458 [DBG] GRPC: [core] Subchannel Connectivity change to SHUTDOWN
2022-03-24 21:28:23.458 [DBG] GRPC: [transport] transport: loopyWriter.run returning. connection error: desc = "transport is closing"
2022-03-24 21:28:23.458 [ERR] LITD: Could not start subservers: unable to start account manager: unable to resume account [REDACTED]: unable to subscribe for account updates: checking pending batch failed: error removing pending batch artifacts: error abandoning channels from previous pending batch: error locating channel outpoint: no channel output found in batch tx for matched order [REDACTED]
2022-03-24 21:28:23.458 [INF] LOOPD: Stopping gRPC server
2022-03-24 21:28:23.458 [INF] LOOPD: Stopping REST server
2022-03-24 21:28:23.459 [INF] LOOPD: Liquidity manager stopped
Did you move or rename your pool.db? Pool should start up fine after doing that.
Ah, I just saw that removing the pool DB was only mentioned in the point above the one I linked to. But in your case you should follow the "Pool account is not synced" guide.
@guggero Thanks! Renaming my pool.db and restarting the litd worked. Then I was able to recover the account. When I attempt to create an ask order I get the following message:
[ERR] AUCT: [/poolrpc.ChannelAuctioneer/SubmitOrder]: rpc error: code = Unknown desc = cannot submit channel type preference with old trader client, please update your software
I'm running pool v5.5 as part of lit v6.5
I would say that the original issue is fixed. Let me know if I should create a new issue related to this error message or if it should stay as part of this issue.
Good to hear you can start LiT/Pool again.
The error you get is known and a patch is being worked on: lightninglabs/lightning-terminal#332
This only affects the UI, you should be able to submit orders through the CLI. Or you can use Terminal Web as a UI to submit orders.
Closing the issue because the original problem seems to be solved.