Redis spike issue
Pradeep205 opened this issue · 14 comments
I'm using Redis db which have capacity of 1gm and I stored 10mb data in one key with json, one of my api is accessing that key 10hits per second, it is spiking suddenly throwing memory outage issue, causing spike, any leads?
@Pradeep205 which command did you use?
can you give the INFO ALL?
get command to access the data from redis
@Pradeep205 please also share the output of INFO ALL, thanks.
Server
redis_version:5.0.6
redis_git_sha1:0
redis_git_dirty:0
redis_build_id:0
redis_mode:standalone
os:Amazon ElastiCache
arch_bits:64
multiplexing_api:epoll
atomicvar_api:atomic-builtin
gcc_version:0.0.0
process_id:1
run_id:bb9c428f7fc71a2b8613b07225d1b1d7270a7835
tcp_port:6379
uptime_in_seconds:10300
uptime_in_days:0
hz:10
configured_hz:10
lru_clock:9758314
executable:-
config_file:-
Clients
connected_clients:6
client_recent_max_input_buffer:4
client_recent_max_output_buffer:0
blocked_clients:0
Memory
used_memory:17276424
used_memory_human:16.48M
used_memory_rss:19603456
used_memory_rss_human:18.70M
used_memory_peak:29879376
used_memory_peak_human:28.50M
used_memory_peak_perc:57.82%
used_memory_overhead:4661320
used_memory_startup:4525496
used_memory_dataset:12615104
used_memory_dataset_perc:98.93%
allocator_allocated:17767824
allocator_active:18276352
allocator_resident:38113280
used_memory_lua:37888
used_memory_lua_human:37.00K
used_memory_scripts:0
used_memory_scripts_human:0B
number_of_cached_scripts:0
maxmemory:1103269725
maxmemory_human:1.03G
maxmemory_policy:volatile-lru
allocator_frag_ratio:1.03
allocator_frag_bytes:508528
allocator_rss_ratio:2.09
allocator_rss_bytes:19836928
rss_overhead_ratio:0.51
rss_overhead_bytes:-18509824
mem_fragmentation_ratio:1.14
mem_fragmentation_bytes:2368928
mem_not_counted_for_evict:0
mem_replication_backlog:0
mem_clients_slaves:0
mem_clients_normal:135744
mem_aof_buffer:0
mem_allocator:jemalloc-5.1.0
active_defrag_running:0
lazyfree_pending_objects:0
Persistence
loading:0
rdb_changes_since_last_save:1
rdb_bgsave_in_progress:0
rdb_last_save_time:1721024046
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:-1
rdb_current_bgsave_time_sec:-1
rdb_last_cow_size:0
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
aof_last_cow_size:0
Stats
total_connections_received:778
total_commands_processed:29962
instantaneous_ops_per_sec:3
total_net_input_bytes:12050753
total_net_output_bytes:109533498
instantaneous_input_kbps:0.08
instantaneous_output_kbps:12.15
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:0
expired_stale_perc:0.00
expired_time_cap_reached_count:0
evicted_keys:0
keyspace_hits:0
keyspace_misses:72
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:0
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0
Replication
role:master
connected_slaves:0
master_replid:--
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:0
second_repl_offset:-1
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0
CPU
used_cpu_sys:6.126003
used_cpu_user:9.802325
used_cpu_sys_children:0.000000
used_cpu_user_children:0.000000
Commandstats
cmdstat_ping:calls=16323,usec=13959,usec_per_call=0.86
cmdstat_command:calls=4,usec=1270,usec_per_call=317.50
cmdstat_client:calls=346,usec=9035,usec_per_call=26.11
cmdstat_select:calls=74,usec=78,usec_per_call=1.05
cmdstat_set:calls=1,usec=6,usec_per_call=6.00
cmdstat_exists:calls=72,usec=111,usec_per_call=1.54
cmdstat_info:calls=12812,usec=862824,usec_per_call=67.34
cmdstat_config:calls=330,usec=6804,usec_per_call=20.62
SSL
ssl_enabled:no
ssl_connections_to_previous_certificate:0
ssl_connections_to_current_certificate:0
ssl_current_certificate_not_before_date:(null)
ssl_current_certificate_not_after_date:(null)
ssl_current_certificate_serial:0
Cluster
cluster_enabled:0
Keyspace
db12:keys=1,expires=0,avg_ttl=0
Any update @sundb
why is used_memory_peak_human only 28.5M? how do you get spiking?
@sundb This is current state I shared, when it started spiking I switched to MySQL, So it came down, It was reached 1GB when redis was connected
but used_memory_peak_human doesn't decrese over time, it's always the pick during running.
if DB have only one key and have 10mb data only, how can memory increase if multiple user are accessing, want to understand how it works @sundb
used_memory_human is 16.48M, when you execute the GET command for the 10mb key, the reply buffer will consume 10mb memory, 16.48 + 10 is close to 28.50.
If multiple user commands arrive in the same event loop, the spike will be higher.
Users are different and we are closing after every GET
it the commnds are one by one, the spkie should be always 28.5mb.
That is what was not able to understand why it reached MAX limit and caused spike and burnt out
since I don't know where you know the max limit is reached, but from info it's not, maybe you can raise an issue with AWS.