Undesirable "hang" behavior on startup if blockchain JSON/RPC endpoint unavailable

[peterbroadhurst]

If EVMConnect cannot talk to the blockchain, then it blocks startup of the FFTM HTTP interface.
This is "as designed" in terms of it's waiting to restore all the existing listeners before it brings up the HTTP endpoint.
But it's not very pleasant as it behaves as if the whole runtime is hung.

I suggest in hindsight, it's better to allow the start of the event streams to be asynchronous.
To split the restore into two parts:

1. Loading the streams
2. Starting them actively consuming events

goroutine 29 [sleep]:
time.Sleep(0x5f5e1000)
        /usr/local/go/src/runtime/time.go:195 +0x135
github.com/hyperledger/firefly-common/pkg/retry.(*Retry).Do(0xc00051dee0, {0x1098040, 0xc000593ec0}, {0xe928df, 0x18}, 0xc0005d7d60)
        /go/pkg/mod/github.com/hyperledger/firefly-common@v1.2.18/pkg/retry/retry.go:88 +0xa5
github.com/hyperledger/firefly-evmconnect/internal/ethereum.(*blockListener).establishBlockHeightWithRetry(0xc0001555a8?)
        /go/pkg/mod/github.com/hyperledger/firefly-evmconnect@v1.3.0/internal/ethereum/blocklistener.go:76 +0x53
github.com/hyperledger/firefly-evmconnect/internal/ethereum.(*blockListener).listenLoop(0xc00008bd40)
        /go/pkg/mod/github.com/hyperledger/firefly-evmconnect@v1.3.0/internal/ethereum/blocklistener.go:93 +0x85
created by github.com/hyperledger/firefly-evmconnect/internal/ethereum.(*blockListener).checkStartedLocked
        /go/pkg/mod/github.com/hyperledger/firefly-evmconnect@v1.3.0/internal/ethereum/blocklistener.go:396 +0x96`

Blocks

goroutine 1 [select]:
github.com/hyperledger/firefly-evmconnect/internal/ethereum.(*blockListener).getHighestBlock(0xc00008bd40, {0x1097f98, 0xc0001c6f50})
        /go/pkg/mod/github.com/hyperledger/firefly-evmconnect@v1.3.0/internal/ethereum/blocklistener.go:414 +0x119
github.com/hyperledger/firefly-evmconnect/internal/ethereum.(*ethConnector).EventStreamStart(0xc000136880, {0x1097f98, 0xc0001c6f50}, 0xc00038a0c0)
        /go/pkg/mod/github.com/hyperledger/firefly-evmconnect@v1.3.0/internal/ethereum/event_actions.go:47 +0x22c
github.com/hyperledger/firefly-transaction-manager/internal/events.(*eventStream).Start(0xc000513b80, {0x1097f98, 0xc0001c6410})
        /go/pkg/mod/github.com/hyperledger/firefly-transaction-manager@v1.3.4-0.20230908160043-67a5118841aa/internal/events/eventstream.go:506 +0x6d9
github.com/hyperledger/firefly-transaction-manager/pkg/fftm.(*manager).restoreStreams(0xc0004520f0)
        /go/pkg/mod/github.com/hyperledger/firefly-transaction-manager@v1.3.4-0.20230908160043-67a5118841aa/pkg/fftm/stream_management.go:63 +0x1eb
github.com/hyperledger/firefly-transaction-manager/pkg/fftm.(*manager).Start(0xc0004520f0)
        /go/pkg/mod/github.com/hyperledger/firefly-transaction-manager@v1.3.4-0.20230908160043-67a5118841aa/pkg/fftm/manager.go:171 +0xc5
github.com/hyperledger/firefly-evmconnect/cmd.runManager({0x1098040, 0xc000037b90}, {0x1094918, 0xc0004520f0})
        /go/pkg/mod/github.com/hyperledger/firefly-evmconnect@v1.3.0/cmd/evmconnect.go:119 +0x31
github.com/hyperledger/firefly-evmconnect/cmd.run()
        /go/pkg/mod/github.com/hyperledger/firefly-evmconnect@v1.3.0/cmd/evmconnect.go:115 +0x335
github.com/hyperledger/firefly-evmconnect/cmd.glob..func1(0x1658600?, {0xe7c05f?, 0x2?, 0x2?})
        /go/pkg/mod/github.com/hyperledger/firefly-evmconnect@v1.3.0/cmd/evmconnect.go:45 +0x17
github.com/spf13/cobra.(*Command).execute(0x1658600, {0xc00019a160, 0x2, 0x2})
        /go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:916 +0x862
github.com/spf13/cobra.(*Command).ExecuteC(0x1658600)
        /go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:1044 +0x3bd
github.com/spf13/cobra.(*Command).Execute(...)
        /go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:968
github.com/hyperledger/firefly-evmconnect/cmd.Execute(...)
        /go/pkg/mod/github.com/hyperledger/firefly-evmconnect@v1.3.0/cmd/evmconnect.go:69
main.main()
        /evmconnect/evmconnect/main.go:20 +0x51