Disconnect causing hard-to-diagnose error
greim opened this issue · 7 comments
This line
Line 63 in 2ef6dee
In other words, an error
event emitted by the redis client is causing koa-redis to emit a disconnect
event, which in turn causes koa-generic-session to have this error:
error: Error: session store error
at Object.getSession (/path/to/node_modules/koa-generic-session/lib/session.js:122:13)
at GeneratorFunctionPrototype.next (native)
at Object.session (/path/to/node_modules/koa-generic-session/lib/session.js:221:36)
at GeneratorFunctionPrototype.next (native)
at next (/path/to/node_modules/koa/node_modules/co/index.js:74:21)
at Object.<anonymous> (/path/to/node_modules/koa/node_modules/co/index.js:45:5)
at Object.next (/path/to/node_modules/koa/node_modules/co/index.js:90:21)
at Object.<anonymous> (/path/to/node_modules/koa/node_modules/co/index.js:93:18)
at Immediate._onImmediate (/path/to/node_modules/koa/node_modules/co/index.js:52:14)
at processImmediate [as _immediateCallback] (timers.js:374:17)
Which is a bit vague; especially hard since this error seems to be environment-dependent and intermittent.
Are you sure you're running the redis daemon in your machine?!
Yes; it would be immediately obvious if it wasn't running. Rather, this error gets dumped to logs very occasionally/intermittently as the app runs, plus we only see it in production. That—and the error message being not very informative—is why it's hard to diagnose. I've been adding some experimental code to try to better capture the errors directly on the redis client, but the issue is so intermittent that I haven't had results yet. I'll post back here if I find anything relevant.
@greim Any luck pinpointing the issue?
Things have been quiet on this front lately.
Okay there's definitely still errors happening pretty frequently.
client.on('error', this.emit.bind(this, 'disconnect'));
client.on('end', this.emit.bind(this, 'disconnect'));
client.on('connect', this.emit.bind(this, 'connect'));
In my original description above, I blamed the error
event, but I added capturing/logging logic and there's no error object, so it's apparently the end
event. Basically it's acting as if the client were randomly end
-ing and then connect
-ing. Every once in a while, by chance, someone hits the server between one of those intervals and it blows up.
I suspect it's some glitch in redis or the redis client and there's nothing koa-redis can do. We were at one point using the redis
module as our client, but then we switched to redis-sentinel
, but the error has been happening all along.
@greim are you still experiencing this issue?
@DaAwesomeP I've moved to another company so I'm out of the loop on this one unfortunately.