Meteor server appears not to come back when redis restarted

[deanrad]

After dying with the following, Meteor appears not to come back (I see it exits and waits for a file change).
Is there a good way to add some reconnect logic, or otherwise provide some fault tolerance? Note, that I was testing this scenario with keyspace notifications enabled, intending to test out the usecase of keyspace notifications surviving a restart of a persisted redis server.

Thanks for a great package that works pretty damn well out of the box - the keyspace notifications are the best!

09:03:17 web.1 |  W20141104-09:03:17.731(-6)? (STDERR)
09:03:17 web.1 |  W20141104-09:03:17.733(-6)? (STDERR) events.js:72
09:03:17 web.1 |  W20141104-09:03:17.733(-6)? (STDERR)         throw er; // Unhandled 'error' event
09:03:17 web.1 |  W20141104-09:03:17.734(-6)? (STDERR)               ^
09:03:17 web.1 |  W20141104-09:03:17.734(-6)? (STDERR) Error: Redis connection to localhost:6379 failed - connect ECONNREFUSED
09:03:17 web.1 |  W20141104-09:03:17.735(-6)? (STDERR)     at RedisClient.on_error (/Users/dradcliffe/.meteor/packages/slava:redis-livedata/.1.0.0.3iy2q3++o…
09:03:17 web.1 |  W20141104-09:03:17.735(-6)? (STDERR)     at Socket.<anonymous> (/Users/dradcliffe/.meteor/packages/slava:redis-livedata/.1.0.0.3iy2q3++os+…
09:03:17 web.1 |  W20141104-09:03:17.735(-6)? (STDERR)     at Socket.emit (events.js:95:17)
09:03:17 web.1 |  W20141104-09:03:17.735(-6)? (STDERR)     at net.js:440:14
09:03:17 web.1 |  W20141104-09:03:17.735(-6)? (STDERR)     at process._tickCallback (node.js:419:13)
09:03:17 web.1 |  => Exited with code: 8

[deanrad]

Just seeing if there's any progress on this - I might not be able to use this if the library kills the process it is in when Redis momentarily goes away.

[Slava]

@chicagogrooves when does it happen? Why does the redis server restart?

[deanrad]

In development, I kill the Redis server on purpose, to play the role of chaos monkey. I've been told by ops to keep an eye on the behavior of all our apps in the face of dependent service restarts.

[Slava]

@chicagogrooves There are two ways to go about it:

• implement a Redis-node client reconnect on failure (you would need to decide how many times to retry and when to give up and how to report this, I would review a PR for that)
• restart your app server when it crashes, the same way you do it with your Redis servers. If you have ops infra in place to handle the Redis failures, you probably have one for your app servers, too.

[deanrad]

Cool, @Slava thanks - I'll close this.. Are you maintaining a FAQ or wiki?

[Slava]

@chicagogrooves no, I don't maintain any wikis related to redis at this point.