koajs/koa-redis

Disconnect causing hard-to-diagnose error

greim opened this issue · 7 comments

This line

client.on('error', this.emit.bind(this, 'disconnect'));
is causing an error https://github.com/koajs/generic-session/blob/55429585d7d3219f3a7acdef90794456ad670b2d/lib/session.js#L122

In other words, an error event emitted by the redis client is causing koa-redis to emit a disconnect event, which in turn causes koa-generic-session to have this error:

error: Error: session store error
   at Object.getSession (/path/to/node_modules/koa-generic-session/lib/session.js:122:13)
   at GeneratorFunctionPrototype.next (native)
   at Object.session (/path/to/node_modules/koa-generic-session/lib/session.js:221:36)
   at GeneratorFunctionPrototype.next (native)
   at next (/path/to/node_modules/koa/node_modules/co/index.js:74:21)
   at Object.<anonymous> (/path/to/node_modules/koa/node_modules/co/index.js:45:5)
   at Object.next (/path/to/node_modules/koa/node_modules/co/index.js:90:21)
   at Object.<anonymous> (/path/to/node_modules/koa/node_modules/co/index.js:93:18)
   at Immediate._onImmediate (/path/to/node_modules/koa/node_modules/co/index.js:52:14)
   at processImmediate [as _immediateCallback] (timers.js:374:17)

Which is a bit vague; especially hard since this error seems to be environment-dependent and intermittent.

Are you sure you're running the redis daemon in your machine?!

Yes; it would be immediately obvious if it wasn't running. Rather, this error gets dumped to logs very occasionally/intermittently as the app runs, plus we only see it in production. That—and the error message being not very informative—is why it's hard to diagnose. I've been adding some experimental code to try to better capture the errors directly on the redis client, but the issue is so intermittent that I haven't had results yet. I'll post back here if I find anything relevant.

@greim Any luck pinpointing the issue?

Things have been quiet on this front lately.

Okay there's definitely still errors happening pretty frequently.

  client.on('error', this.emit.bind(this, 'disconnect'));
  client.on('end', this.emit.bind(this, 'disconnect'));
  client.on('connect', this.emit.bind(this, 'connect'));

In my original description above, I blamed the error event, but I added capturing/logging logic and there's no error object, so it's apparently the end event. Basically it's acting as if the client were randomly end-ing and then connect-ing. Every once in a while, by chance, someone hits the server between one of those intervals and it blows up.

I suspect it's some glitch in redis or the redis client and there's nothing koa-redis can do. We were at one point using the redis module as our client, but then we switched to redis-sentinel, but the error has been happening all along.

@greim are you still experiencing this issue?

greim commented

@DaAwesomeP I've moved to another company so I'm out of the loop on this one unfortunately.