Backend service crashes after "Peer.outHandler: Problem sending message to peer"
MirekR opened this issue · 6 comments
Hi, we've been experience strange behavior of the node - it seems to be loosing connection to one of it's peers resulting in crash of the backend container - see the logs attached. Not sure if that's related but also from time to time we can see huge consumption of the memory by the backend (up 99.5%) causing unresponsiveness of the system.
Dockers are run in the Google Cloud VM, 6CPU and 32BG
failed.log
Are you running out of disk space? I'm seeing a lot of weird errors in that log
At the moment, disk usage is at 88%, meaning 8.5GB is still free.
I'd recommend increasing your disk size and also making sure ulimit -n is set to a number higher than default. Please let me know if that doesn't help and I'm happy to reopen this issue.
Hi, it looks like server is not falling over after increasing disk quota but unfortunately doesn't behave correctly either. It felt out of sync with the logs attached. It almost feels these two issues are connected as you can see mention of lost connections.
Do you think it's setup or there is fundamental issue with the backend server and how it handles disconnecting servers / how it selecting peers?
Can you provide the complete log output? Something else failed earlier up in the logs
Unfortunately I purged the server to get it working again. I'll get full logs when it happens next time.