pnxtech/hydra

Nodejs crashes when redis connection dies - How to handle it ?

FDT2k opened this issue · 1 comments

FDT2k commented

Hi,

I'm considering using hydra for a small project but I have a big problem. When I restart redis, all the related microservices crashes, without retrying the connection

I tried to tweak the retry_strategy but I'm a bit lost.

How can I catch theses exceptions or prevent them ?

log entry { ts: '2018-06-13T09:36:04.251Z',
serviceName: 'my-awesome-microservice',
type: 'error',
processID: 1953,
msg: 'Established Redis server connection has closed' }
Unhandled Exception { Error: Redis connection to 127.0.0.1:6379 failed - connect ECONNREFUSED 127.0.0.1:6379
at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1174:14)
errno: 'ECONNREFUSED',
code: 'ECONNREFUSED',
syscall: 'connect',
address: '127.0.0.1',
port: 6379 }
Unhandled Exception { Error: Redis connection to 127.0.0.1:6379 failed - connect ECONNREFUSED 127.0.0.1:6379
at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1174:14)
errno: 'ECONNREFUSED',
code: 'ECONNREFUSED',
syscall: 'connect',
address: '127.0.0.1',
port: 6379 }
log entry { ts: '2018-06-13T09:36:04.458Z',
serviceName: 'my-awesome-microservice',
type: 'error',
processID: 1953,
msg: 'Reconnecting to Redis server...' }
Error: Redis connection to 127.0.0.1:6379 failed - connect ECONNREFUSED 127.0.0.1:6379
at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1174:14)

Thank you in advance.

cjus commented

@FDT2k Yes and that's by design. In a containerization environment (we use Docker Swarm) a Restart of Redis would cause services to restart. There is a retry period in Hydra which is supposed to retry connecting to Redis for a period. We are aware that this restart logic needs some work. PRs and feedback are definitely welcomed!