RedisLabs/redis-cluster-proxy

Redis Cluster Proxy Crash When I write redis data in batches

JackDan9 opened this issue · 0 comments

[2023-03-03 11:46:20.643/M] Redis Cluster Proxy v999.999.999 (unstable)
[2023-03-03 11:46:20.643/M] Commit: (ac83840/1)
[2023-03-03 11:46:20.643/M] Git Branch: unstable
[2023-03-03 11:46:20.643/M] PID: 1
[2023-03-03 11:46:20.643/M] OS: Linux 3.10.0-957.el7.x86_64 x86_64
[2023-03-03 11:46:20.643/M] Bits: 64
[2023-03-03 11:46:20.643/M] Log level: error
[2023-03-03 11:46:20.643/M] Connections pool size: 10 (respawn 2 every 50ms if below 10)
[2023-03-03 11:46:20.644/M] Starting 8 threads...
[2023-03-03 11:46:20.644/M] Fetching cluster configuration...
[2023-03-03 11:46:20.647/M] Cluster Address: redis-cluster:6379
[2023-03-03 11:46:20.647/M] Cluster has 3 masters and 3 replica(s)
[2023-03-03 11:46:20.731/M] All thread(s) started!
[2023-03-03 11:49:55.003/6] Thread 0 terminated
[2023-03-03 11:49:55.003/6] Thread 1 terminated
[2023-03-03 11:49:55.003/6] Thread 2 terminated
[2023-03-03 11:49:55.003/6] Thread 3 terminated
[2023-03-03 11:49:55.003/6] Thread 4 terminated
[2023-03-03 11:49:55.003/6] Thread 5 terminated
[2023-03-03 11:49:55.003/6] Thread 7 terminated

=== PROXY BUG REPORT START: Cut & paste starting from here ===
[2023-03-03 11:49:55.003/6] Redis Cluster Proxy 999.999.999 crashed by signal: 11
[2023-03-03 11:49:55.003/6] Crashed running the instruction at: 0x40831c
[2023-03-03 11:49:55.003/6] Accessing address: 0x1c0
[2023-03-03 11:49:55.003/6] Handling crash on thread: 6

------ STACK TRACE ------
EIP:
redis-cluster-proxy(listDelNode+0xc)[0x40831c]

Backtrace:
redis-cluster-proxy(logStackTrace+0x2d)[0x40d0fd]
redis-cluster-proxy(sigsegvHandler+0x186)[0x40d716]
/lib64/libpthread.so.0(+0xf630)[0x7f1538239630]
redis-cluster-proxy(listDelNode+0xc)[0x40831c]
redis-cluster-proxy[0x410d2f]
redis-cluster-proxy[0x419d88]
redis-cluster-proxy(aeProcessEvents+0x13b)[0x408d7b]
redis-cluster-proxy(aeMain+0x2b)[0x40913b]
redis-cluster-proxy[0x410f9c]
/lib64/libpthread.so.0(+0x7ea5)[0x7f1538231ea5]
/lib64/libc.so.6(clone+0x6d)[0x7f1537f5a96d]

------ INFO OUTPUT ------

Proxy

proxy_version:999.999.999
proxy_git_sha1:ac83840d
proxy_git_dirty:1
proxy_git_branch:unstable
os:Linux 3.10.0-957.el7.x86_64 x86_64
arch_bits:64
multiplexing_api:epoll
gcc_version:7.3.1
process_id:1
threads:8
tcp_port:7777
uptime_in_seconds:215
uptime_in_days:0
config_file:/data/proxy.conf
acl_user:default

Memory

used_memory:17357512
used_memory_human:16.55M
total_system_memory:405186953216
total_system_memory_human:377.36G

Clients

connected_clients:6
max_clients:10000
thread_0_clinets:1
thread_1_clinets:1
thread_2_clinets:1
thread_3_clinets:0
thread_4_clinets:1
thread_5_clinets:1
thread_6_clinets:1
thread_7_clinets:0

Cluster

address:
entry_node::0

---- SIZEOF STRUCTS ----
clientRequest: 184
client: 224
redisClusterConnection: 48
clusterNode: 112
redisCluster: 104
list: 48
listNode: 24
rax: 24
raxNode: 4
raxIterator: 480
aeEventLoop: 88
aeFileEvent: 32
aeTimeEvent: 64

------ REGISTERS ------

RAX:000000000109ae70 RBX:00000000000001c0
RCX:00007f15382237b8 RDX:0000000000000006
RDI:000000000109aeb0 RSI:00000000000001c0
RBP:000000000109aeb0 RSP:00007f15347cddd0
R8 :0000000000000046 R9 :000000000042be54
R10:0000000000000004 R11:373736310a0d3031
R12:0000000000000000 R13:000000000109aeb0
R14:000000000108adc0 R15:00000000010acfa0
RIP:000000000040831c EFL:0000000000010202
CSGSFS:0000000000000033
(00007f15347cdddf) -> 00007f1518261570
(00007f15347cddde) -> 0000000000000001
(00007f15347cdddd) -> 0000000000000143
(00007f15347cdddc) -> 00000000010acfa4
(00007f15347cdddb) -> 0000000000000143
(00007f15347cddda) -> 000000000118d3f0
(00007f15347cddd9) -> 000000061825ee60
(00007f15347cddd8) -> 0000000000000000
(00007f15347cddd7) -> 0000000000419ea8
(00007f15347cddd6) -> 000000000042b45d
(00007f15347cddd5) -> 0000000000419d88
(00007f15347cddd4) -> 00007f151818e6a0
(00007f15347cddd3) -> 0000000000410d2f
(00007f15347cddd2) -> 00007f1518261570
(00007f15347cddd1) -> 00007f151818e6a0
(00007f15347cddd0) -> 00007f1518261570

------ DUMPING CODE AROUND EIP ------
Symbol: listDelNode (base: 0x408310)
Module: redis-cluster-proxy (base 0x400000)
$ xxd -r -p /tmp/dump.hex /tmp/dump.bin
$ objdump --adjust-vma=0x408310 -D -b binary -m i386:x86-64 /tmp/dump.bin

dump of function (hexdump of 140 bytes):
55534889fd4889f34883ec08488b16488b46084885d2743848894208488b46084885c07433488910488b45184885c07406488b7b10ffd04889dfe801ad010048836d28014883c4085b5dc30f1f4400004885c048890775cd48895508ebca669055534889fdbf1000000089f34883ec08e8bbab01004885c0740e85db7412488b55084889108958084883c408
Function at 0x423050 is zfree
Function at 0x422f40 is zmalloc

=== PROXY BUG REPORT END. Make sure to include from START to END. ===

   Please report the crash by opening an issue on github:

       https://github.com/artix75/redis-cluster-proxy/issues