connect ETIMEDOUT 172.17.0.6:7000
sparachi opened this issue · 11 comments
I wrote an node api using thunk-redis, and testing from POSTman. The data get or set works sometimes and sometimes it just hangs in there, and times out with the error message "connect ETIMEDOUT 172.17.0.6:7000"
I followed steps here in https://github.com/Grokzen/docker-redis-cluster to create a docker image with 7000 -> 7002 exposed for clients.
The documentation is not so helpful w.r.t cluster configuration. How do I debug and isolate the issue?
Could you connect redis with redis-cli ?
Yes I can connect to using cli. I can connect from the node app also, but the connection is not reliable. It hangs couple of times, and does not return anything.
@sparachi I think that this option killed the client. cluster-node-timeout 5000 https://github.com/Grokzen/docker-redis-cluster/blob/master/docker-data/redis-conf/7000/redis.conf#L4
I have removed the entry in the redis.conf and restarted the docker container. But the issue still persists.
@sparachi I got the problem: redis internal IP and external IP are different.
when connect:
const cli = redis.createClient([
"127.0.0.1:7000",
"127.0.0.1:7001",
"127.0.0.1:7002",
"127.0.0.1:7003",
"127.0.0.1:7004",
"127.0.0.1:7005",
"127.0.0.1:7006",
"127.0.0.1:7007"
])but cluster slots command give me:
[ [ 0, 5460, [ '172.17.0.2', 7000 ], [ '172.17.0.2', 7003 ] ],
[ 10923, 16383, [ '172.17.0.2', 7002 ], [ '172.17.0.2', 7005 ] ],
[ 5461, 10922, [ '172.17.0.2', 7001 ], [ '172.17.0.2', 7004 ] ] ]
that means:
If we send a command client.get("somekey") on connection 127.0.0.1:7000, redis cannot find the key on 7000, and respond a error (error) MOVED 9842 172.17.0.2:7001, then client try to connect to 172.17.0.2:7001, but it is internal IP!
You can try this with redis-cli, look like:
127.0.0.1:7000> get 1
(error) MOVED 9842 172.17.0.2:7001
I can resolve it with a new option IPMap
@zensh I still the see the same behavior. Below is my modified createClient method.
I exposed all the 6 ports
docker run -d --name redis-cluster -p 7000:7000 -p 7001:7001 -p 7002:7002 -p 7003:7003 -p 7004:7004 -p 7005:7005 redis-cluster
(Ideally I think I need to do writes on master i.e. 7000 - 7002, reads on slaves 7003-7005. But for now in dev I exposed all the 6 ports since all are running on the same instance of docker container)
let clusterNodes = [
'127.0.0.1:7000',
'127.0.0.1:7001',
'127.0.0.1:7002',
'127.0.0.1:7003',
'127.0.0.1:7004',
'127.0.0.1:7005'
];
let options = {
clusterMode: true,
noDelay: true,
usePromise: blueBird,
IPMap: {
'172.17.0.7:7000': '127.0.0.1:7000',
'172.17.0.7:7001': '127.0.0.1:7001',
'172.17.0.7:7002': '127.0.0.1:7002',
'172.17.0.7:7003': '127.0.0.1:7003',
'172.17.0.7:7004': '127.0.0.1:7004',
'172.17.0.7:7005': '127.0.0.1:7005'
}
};
let redisClient = redis.createClient(clusterNodes, options);
In my GET method I do this
return new Promise(function(resolve, reject) {
// id is the parameter passed to this method
let data = {};
data.key = JSON.parse(id);
console.log("value to query " + data.key);
redisClient.get(data.key).then(function (res) {
console.log('data get from redis:', res);
if(res){
data.response = res;
} else {
data.response = 'null';
}
return redisClient.quit();
})
.then(function (res) {
console.log('redis client quit:', res);
// redis client quit: OK
return resolve(data);
})
.catch(function (err) {
console.error(err);
return reject(err);
});
});
In my SET method I do this below
return new Promise(function(resolve, reject) {
let data = {"status": "success"};
redisClient.set(dataReceived.key, dataReceived.value)
.then(function (res) {
console.log('data set to redis: ', res);
return redisClient.get(dataReceived.key);
})
.then(function (res) {
console.log('data get from redis:', res);
return redisClient.quit();
})
.then(function (res) {
console.log('redis client quit:', res);
// redis client quit: OK
return resolve(data);
})
.catch(function (err) {
console.error(err);
return reject(err);
});
});
Even though if it works, I have some questions around the production setup.
- do I need to mention all the docker ip's, as well as EC2 host ip's when I createClient() in my node API. Since all my redis nodes dockerized, node api dockerized will be running in different EC2 instances and they will have dynamic IP's.
- did someone try this in production deployment as you know of
@sparachi Currently no way to announce different IP/port. redis/redis#2704
redis/redis#2527 (comment)
says it is implemented. I am confused here.