Replication is not working from appication/client side

Question

Replication is not working from appication/client side

Dwijad opened this issue 3 years ago · 1 comments

Hi
I have three node dynomite cluster setup in VMWARE along with Redis server running in each of those three nodes. The replication is working fine if i use redis-cli. However one of the application that i am trying to integrate with dynomite is unable to read/write to the cluster.

Dynomite Version: dynomite-v0.6.21-rc2-19-gb56cf6a
Redis version: 6.2.6
redis-cli 6.2.6
CentOS 7

Node 1: 172.16.10.242

dyn_o_mite:
  datacenter: dc-a
  rack: rack1
  dyn_listen: 0.0.0.0:7379
  dyn_seed_provider: simple_provider
  dyn_seeds:
  - 172.16.10.48:7379:rack1:dc-b:0
  - 172.16.10.49:7379:rack1:dc-c:0
  listen: 172.16.10.242:8379
  servers:
  - 0.0.0.0:6379:1
  tokens: '0'
  secure_server_option: datacenter
  pem_key_file: /root/dynomite/conf/dynomite.pem
  data_store: 0
  stats_listen: 172.16.10.242:22222
  read_consistency : DC_QUORUM
  write_consistency : DC_QUORUM

Node 2: 172.16.10.48

dyn_o_mite:
  datacenter: dc-b
  rack: rack1
  dyn_seed_provider: simple_provider
  dyn_listen: 0.0.0.0:7379
  dyn_seeds:
  - 172.16.10.49:7379:rack1:dc-c:0
  - 172.16.10.242:7379:rack1:dc-a:0
  listen: 172.16.10.48:8379
  servers:
  - 0.0.0.0:6379:1
  tokens: '0'
  secure_server_option: datacenter
  pem_key_file: /root/dynomite/conf/dynomite.pem
  data_store: 0
  stats_listen: 172.16.10.48:22222
  read_consistency : DC_QUORUM
  write_consistency : DC_QUORUM

Node 3: 172.16.10.48

dyn_o_mite:
  datacenter: dc-c
  rack: rack1
  dyn_listen: 0.0.0.0:7379
  dyn_seed_provider: simple_provider
  dyn_seeds:
  - 172.16.10.242:7379:rack1:dc-a:0
  - 172.16.10.48:7379:rack1:dc-b:0
  listen: 172.16.10.49:8379
  servers:
  - 0.0.0.0:6379:1
  tokens: '0'
  secure_server_option: datacenter
  pem_key_file: /root/dynomite/conf/dynomite.pem
  data_store: 0
  stats_listen: 172.16.10.49:22222
  read_consistency : DC_QUORUM
  write_consistency : DC_QUORUM

The replication is working fine if i use the redis-cli

[root@redis ~]# redis-cli -h 172.16.10.242 -p 8379
172.16.10.242:8379> SET C 1
OK

[root@redis-2 ~]# redis-cli -h 172.16.10.48 -p 8379
172.16.10.48:8379> 
172.16.10.48:8379> KEYS *
1) "V"
2) "X"
3) "C"

However if i use my application to write to one of the node(172.16.10.242:8379) of dynomite cluster, i am getting following error:

[2022-04-23 10:04:37.154] redis_parse_req:2387 parsed bad req 2980 res 1 type 0 state 5
00000000  2a 31 0d 0a 24 35 0d 0a  4d 55 4c 54 49 0d 0a 2a   |*1..$5..MULTI..*|
00000010  32 0d 0a 24 34 0d 0a 4c  4c 45 4e 0d 0a 24 35 36   |2..$4..LLEN..$56|
00000020  0d 0a 33 34 33 64 35 37  30 33 2d 66 30 65 32 2d   |..343d5703-f0e2-|
00000030  33 66 33 64 2d 39 64 39  36 2d 30 31 65 35 38 37   |3f3d-9d96-01e587|
00000040  61 32 37 30 30 33 2e 72  65 70 6c 79 2e 63 65 6c   |a27003.reply.cel|
00000050  65 72 79 2e 70 69 64 62  6f 78 0d 0a 2a 32 0d 0a   |ery.pidbox..*2..|
00000060  24 34 0d 0a 4c 4c 45 4e  0d 0a 24 35 39 0d 0a 33   |$4..LLEN..$59..3|
00000070  34 33 64 35 37 30 33 2d  66 30 65 32 2d 33 66 33   |43d5703-f0e2-3f3|
00000080  64 2d 39 64 39 36 2d 30  31 65 35 38 37 61 32 37   |d-9d96-01e587a27|
00000090  30 30 33 2e 72 65 70 6c  79 2e 63 65 6c 65 72 79   |003.reply.celery|
000000a0  2e 70 69 64 62 6f 78 06  16 33 0d 0a 2a 32 0d 0a   |.pidbox..3..*2..|
000000b0  24 34 0d 0a 4c 4c 45 4e  0d 0a 24 35 39 0d 0a 33   |$4..LLEN..$59..3|
000000c0  34 33 64 35 37 30 33 2d  66 30 65 32 2d 33 66 33   |43d5703-f0e2-3f3|
000000d0  64 2d 39 64 39 36 2d 30  31 65 35 38 37 61 32 37   |d-9d96-01e587a27|
000000e0  30 30 33 2e 72 65 70 6c  79 2e 63 65 6c 65 72 79   |003.reply.celery|
000000f0  2e 70 69 64 62 6f 78 06  16 36 0d 0a 2a 32 0d 0a   |.pidbox..6..*2..|
00000100  24 34 0d 0a 4c 4c 45 4e  0d 0a 24 35 39 0d 0a 33   |$4..LLEN..$59..3|
00000110  34 33 64 35 37 30 33 2d  66 30 65 32 2d 33 66 33   |43d5703-f0e2-3f3|
00000120  64 2d 39 64 39 36 2d 30  31 65 35 38 37 61 32 37   |d-9d96-01e587a27|
00000130  30 30 33 2e 72 65 70 6c  79 2e 63 65 6c 65 72 79   |003.reply.celery|
00000140  2e 70 69 64 62 6f 78 06  16 39 0d 0a 2a 31 0d 0a   |.pidbox..9..*1..|
00000150  24 34 0d 0a 45 58 45 43  0d 0a                     |$4..EXEC..|
[2022-04-23 10:04:37.154] core_close:418 close <CONN_CLIENT 0xa7f920 16 from '172.16.10.215:42820'> on event 00FF eof 0 done 0 rb 685 sb 20: Invalid argument
[2022-04-23 10:04:37.154] client_unref_internal_try_put:97 <CONN_CLIENT 0xa7f920 -1 from '172.16.10.215:42820'> unref owner <POOL 0xa31bb0 'dyn_o_mite'>
[2022-04-23 10:04:37.155] conn_recv_data:366 <CONN_CLIENT 0xa80360 15 from '172.16.10.215:42818'> recv eof rb 28 sb 14
[2022-04-23 10:04:37.155] core_close:418 close <CONN_CLIENT 0xa80360 15 from '172.16.10.215:42818'> on event 00FF eof 1 done 1 rb 28 sb 14  
[2022-04-23 10:04:37.156] client_unref_internal_try_put:97 <CONN_CLIENT 0xa80360 -1 from '172.16.10.215:42818'> unref owner <POOL 0xa31bb0 'dyn_o_mite'>
[2022-04-23 10:04:37.169] proxy_accept:203 <CONN_PROXY 0xa333b0 8 listening on '172.16.10.242:8379'> accepted <CONN_CLIENT 0xa80360 15 from '172.16.10.215:42824'>
[2022-04-23 10:04:37.171] proxy_accept:203 <CONN_PROXY 0xa333b0 8 listening on '172.16.10.242:8379'> accepted <CONN_CLIENT 0xa7f920 16 from '172.16.10.215:42826'>
[2022-04-23 10:04:37.173] conn_recv_data:366 <CONN_CLIENT 0xa54a20 13 from '172.16.10.215:42822'> recv eof rb 28 sb 14
[2022-04-23 10:04:37.173] core_close:418 close <CONN_CLIENT 0xa54a20 13 from '172.16.10.215:42822'> on event 00FF eof 1 done 1 rb 28 sb 14  
[2022-04-23 10:04:37.173] client_unref_internal_try_put:97 <CONN_CLIENT 0xa54a20 -1 from '172.16.10.215:42822'> unref owner <POOL 0xa31bb0 'dyn_o_mite'>
[2022-04-23 10:04:37.185] proxy_accept:203 <CONN_PROXY 0xa333b0 8 listening on '172.16.10.242:8379'> accepted <CONN_CLIENT 0xa54a20 13 from '172.16.10.215:42828'>
[2022-04-23 10:04:37.187] redis_parse_req:1583 parsed unsupported command 'MULTI'
[2022-04-23 10:04:37.187] redis_parse_req:2387 parsed bad req 3007 res 1 type 0 state 5

The error in the application side is:

redis.exceptions.ConnectionError: Connection closed by server.

I can reach the dynomite port from remote:

# nc -vz 172.16.10.242 8379
Ncat: Version 7.50 ( https://nmap.org/ncat )
Ncat: Connected to 172.16.10.242:8379.
Ncat: 0 bytes sent, 0 bytes received in 0.02 seconds.

It looks dynomite cluster is working. However i am not sure why the application is not able to write to the dynomite cluster.

Thanks

Answer 1 · 2022-04-25T04:08:12.000Z

It was an issue within the application which was unable to write to the dynomite. Dynomite is working fine.

Also, It looks dynomite is not updating its codebase from last few years. After exploring alternatives to master-master redis cluster, i have found keydb a better option for my requirements.