redis cluster
devthejo opened this issue · 20 comments
Hello it's Jo (formerly @idetoile, now @devthejo),
I recently tried to use the last version of redis-ui on kubernetes with redis cluster and when I connect, I get an infinite loop like error, and UI is stuck on /main/statistics with cog icon, then after a long time I get Error: The Socket.IO timed out for this request (max 300 seconds) ..., with here is my logs during waiting:
...
2020-12-11 01:00:42 | [P3XRS] [PID: 000007] socket.io connection-connect P3Xid100000000000000000000000000000 cluster connected
2020-12-11 01:00:42 | [P3XRS] [PID: 000007] socket.io connection-connect P3Xid100000000000000000000000000000 cluster connected
2020-12-11 01:00:42 | [P3XRS] [PID: 000007] socket.io connection-connect P3Xid100000000000000000000000000000 cluster connected
2020-12-11 01:00:42 | [P3XRS] [PID: 000007] socket.io connection-connect P3Xid100000000000000000000000000000 cluster connected
...
here is my /settings/.p3xrs-conns.json:
{
"list": [
{
"id": "P3Xid100000000000000000000000000000",
"name": "cluster",
"cluster": true,
"nodes": [
{
"id": "P3Xid000000000000000000000000000000",
"host": "redis-redis-cluster-0.redis-redis-cluster-headless.redis.svc.cluster.local",
"port": 6379,
"password": "4fe1502fc513706d11bb9e05a492406e4836867c323b512d52c5bf7d0d95931b"
},
{
"id": "P3Xid000000000000000000000000000001",
"host": "redis-redis-cluster-1.redis-redis-cluster-headless.redis.svc.cluster.local",
"port": 6379,
"password": "4fe1502fc513706d11bb9e05a492406e4836867c323b512d52c5bf7d0d95931b"
},
{
"id": "P3Xid000000000000000000000000000002",
"host": "redis-redis-cluster-2.redis-redis-cluster-headless.redis.svc.cluster.local",
"port": 6379,
"password": "4fe1502fc513706d11bb9e05a492406e4836867c323b512d52c5bf7d0d95931b"
},
{
"id": "P3Xid000000000000000000000000000003",
"host": "redis-redis-cluster-3.redis-redis-cluster-headless.redis.svc.cluster.local",
"port": 6379,
"password": "4fe1502fc513706d11bb9e05a492406e4836867c323b512d52c5bf7d0d95931b"
},
{
"id": "P3Xid000000000000000000000000000004",
"host": "redis-redis-cluster-4.redis-redis-cluster-headless.redis.svc.cluster.local",
"port": 6379,
"password": "4fe1502fc513706d11bb9e05a492406e4836867c323b512d52c5bf7d0d95931b"
},
{
"id": "P3Xid000000000000000000000000000005",
"host": "redis-redis-cluster-5.redis-redis-cluster-headless.redis.svc.cluster.local",
"port": 6379,
"password": "4fe1502fc513706d11bb9e05a492406e4836867c323b512d52c5bf7d0d95931b"
}
]
}
],
"license": ""
}For extra information I run a bitnami helm chart for redis cluster.
If you have any idea of what's going on, I'll appreciate any help.
Thanks in advance if you have a little time to spend on this.
{
"list": [
{
"id": "P3Xid000000000000000000000000000000",
"host": "redis-redis-cluster-0.redis-redis-cluster-headless.redis.svc.cluster.local",
"port": 6379,
"password": "4fe1502fc513706d11bb9e05a492406e4836867c323b512d52c5bf7d0d95931b",
"name": "cluster",
"cluster": true,
"nodes": [
{
"id": "P3Xid000000000000000000000000000001",
"host": "redis-redis-cluster-1.redis-redis-cluster-headless.redis.svc.cluster.local",
"port": 6379,
"password": "4fe1502fc513706d11bb9e05a492406e4836867c323b512d52c5bf7d0d95931b"
},
{
"id": "P3Xid000000000000000000000000000002",
"host": "redis-redis-cluster-2.redis-redis-cluster-headless.redis.svc.cluster.local",
"port": 6379,
"password": "4fe1502fc513706d11bb9e05a492406e4836867c323b512d52c5bf7d0d95931b"
},
{
"id": "P3Xid000000000000000000000000000003",
"host": "redis-redis-cluster-3.redis-redis-cluster-headless.redis.svc.cluster.local",
"port": 6379,
"password": "4fe1502fc513706d11bb9e05a492406e4836867c323b512d52c5bf7d0d95931b"
},
{
"id": "P3Xid000000000000000000000000000004",
"host": "redis-redis-cluster-4.redis-redis-cluster-headless.redis.svc.cluster.local",
"port": 6379,
"password": "4fe1502fc513706d11bb9e05a492406e4836867c323b512d52c5bf7d0d95931b"
},
{
"id": "P3Xid000000000000000000000000000005",
"host": "redis-redis-cluster-5.redis-redis-cluster-headless.redis.svc.cluster.local",
"port": 6379,
"password": "4fe1502fc513706d11bb9e05a492406e4836867c323b512d52c5bf7d0d95931b"
}
]
}
],
"license": ""
}thanks to answer me, but tried your config, and unfortunately same thing append, same message in logs, average two time per second:
2020-12-11 11:37:02 [P3XRS] [PID: 000007] socket.io connection-connect P3Xid000000000000000000000000000000 cluster connected
2020-12-11 11:37:01 [P3XRS] [PID: 000007] socket.io connection-connect P3Xid000000000000000000000000000000 cluster connected
2020-12-11 11:37:01 [P3XRS] [PID: 000007] socket.io connection-connect P3Xid000000000000000000000000000000 cluster connected
2020-12-11 11:37:00 [P3XRS] [PID: 000007] socket.io connection-connect P3Xid000000000000000000000000000000 cluster connected
2020-12-11 11:36:59 [P3XRS] [PID: 000007] socket.io connection-connect P3Xid000000000000000000000000000000 cluster connected
2020-12-11 11:36:59 [P3XRS] [PID: 000007] socket.io connection-connect P3Xid000000000000000000000000000000 cluster connected
no idea, i cannot reproduce it:
i onyl tested here:
docker run -e "IP=0.0.0.0" -p 7000:7000 -p 7001:7001 -p 7002:7002 -p 7003:7003 -p 7004:7004 -p 7005:7005 grokzen/redis-cluster:latestmy config:
{
"name": "cluster",
"host": "localhost",
"port": 7000,
"id": "P3Xide33c6489b610100086d7e771aa0505",
"cluster": true,
"nodes": [
{
"port": 7001,
"id": "P3Xid8cb66489b61010003cd9a68bfea251",
"host": "localhost"
},
{
"port": 7002,
"id": "P3Xid76086489b6101000fc7cadafab5551",
"host": "localhost"
},
{
"port": 7003,
"id": "P3Xiddfe86489b610100008ecfd210ded0d",
"host": "localhost"
},
{
"port": 7004,
"id": "P3Xid9be96489b61010004c20e896e3fc21",
"host": "localhost"
},
{
"port": 7005,
"id": "P3Xideaba6489b6101000854c631c610879",
"host": "localhost"
}
]
}
works perfect:
cluster:
i would first generate the config in the gui, then copy for kubernetes
i sense, maybe the password is giving an error, but i do not use it, you could check the code you wrote is buggy maybe.
i checked out with password, but it works:
{
"name": "cluster",
"host": "localhost",
"port": 7000,
"id": "P3Xide33c6489b610100086d7e771aa0505",
"cluster": true,
"nodes": [
{
"port": 7001,
"id": "P3Xid8cb66489b61010003cd9a68bfea251",
"host": "localhost",
"password": "1234567890"
},
{
"port": 7002,
"id": "P3Xid76086489b6101000fc7cadafab5551",
"host": "localhost",
"password": "1234567890"
},
{
"port": 7003,
"id": "P3Xiddfe86489b610100008ecfd210ded0d",
"host": "localhost",
"password": "1234567890"
},
{
"port": 7004,
"id": "P3Xid9be96489b61010004c20e896e3fc21",
"host": "localhost",
"password": "1234567890"
},
{
"port": 7005,
"id": "P3Xideaba6489b6101000854c631c610879",
"host": "localhost",
"password": "1234567890"
}
],
"password": "1234567890"
}same result, it works, your code :)
same results with generating the config in gui and using it directly, the strange thing is that when I click "test connection" it says "Redis connection successful"
suspected a navigator problem and tested in chromium instead of firefox but same result
does work via docker as the example shows?
i sense it is a network problem.
what is weird is that your code and still getting this error :)
my redis is not exposed to external network, so I have to run redis-ui in kubernetes, found a similar issue:
https://stackoverflow.com/questions/55778291/why-does-connecting-to-a-cluster-constantly-loop-in-ioredis
I will investigate the network
this guy: @sergeyshaykhullin very good in kubernetes, he is using this app sometimes, maybe he can help you. i doubt it is app error. more of a network problem with kubernetes settings.
when i connect to a cluster, it connects only once, your logs shows multiple connects which is weird so something is happening.
thanks for calling for reinforcements to solve my problem, tried to replace hosts with internal IP of pods but same results
tried to add lazyConnect = true to ioredis cluster options, and now it doesn't loop on connected message in logs, but the UI still stuck on "Connection to Redis..."
i sense it is the network configuration (the environment of the setup, ioredis should work as it is). with a proper networking p3x redis ui works with clusters without any problem. you wrote it, you must have had a cluster setup and it worked. and still, it works via a docker proper configuration etc.
have you tried different REDIS GUI clients? maybe that could show some debug info. dunno.
I finally exposed redis external and tested with RedisDesktopManager and doesn't connect (expect when doing test connection, like with RedisUI) but without any additional informations, it's seem effectively related to redis instance (network or setup) and not to redis-ui. Sorry for wasting your time, and thanks again. I'll continue to investate.
maybe you could investigate the redis cluster instance logs, that should give you what is happening....
found it ! the create-cluster job couldn't authenticate because annotation to inject vault secret was not on good metadata key in my kubernetes manifest. Now it works like a charm ! Awesome ! P3X-REDIS-UI rocks !
Have a good continuation !