Handle Blockchain RPC Failures

[p5150j]

If the RPC node goes down, how does Gateway handle it?

Acceptance Criteria

• All Services react to a down RPC in an expected way
• Good logging when RPC is disconnected

Implementation Options (might be different for different services)

• Have the ability for more than one node and try to failover to a different node
• Wait a timeout and exit the application when it cannot reconnect to the RPC
• Go into a degraded "pending" state where it can still collect data, but doesn't submit them to the chain (content publisher for example)

Runtime Upgrade Test

While running Gateway against testnet, testnet did a runtime upgrade which triggered a restart of the services. Assuming k8 restart, this correctly solves the runtime upgrade issue, but may want more.

[aramikm]

Some details might be helpful.