Wrong hit/miss value on Elasticache 7.0.7
gillg opened this issue · 4 comments
Describe the problem
When I compare INFO Stats keyspace_* with the exporter value I see a problem.
The exporter always displays "0" where INFO is displaying:
keyspace_hits:232692
keyspace_misses:137713
What version of redis_exporter are you running?
Please run redis_exporter --version if you're not sure what version you're running.
redis_instance_info{maxmemory_policy="volatile-lru",os="Amazon ElastiCache",process_id="1",redis_build_id="0",redis_mode="standalone",redis_version="7.0.7",role="slave",run_id="467828b573b92ae61bac60356d0548c9a66c4d3a",tcp_port="6379"} 1
Running the exporter
REDIS_ADDR=redis://xxxxxxxxxx-ro.xxxxxx.ng.0001.xxxxx.cache.amazonaws.com:6379
Screenshots
# HELP redis_keyspace_hits_total keyspace_hits_total metric
# TYPE redis_keyspace_hits_total counter
redis_keyspace_hits_total 0
# HELP redis_keyspace_misses_total keyspace_misses_total metric
# TYPE redis_keyspace_misses_total counter
redis_keyspace_misses_total 0
INFO Stats
# Stats
total_connections_received:32
total_commands_processed:560844
instantaneous_ops_per_sec:0
total_net_input_bytes:68835819
total_net_output_bytes:46645299
total_net_repl_input_bytes:7443712
total_net_repl_output_bytes:7702410
instantaneous_input_kbps:0.08
instantaneous_output_kbps:11.17
instantaneous_input_repl_kbps:0.00
instantaneous_output_repl_kbps:0.03
rejected_connections:0
sync_full:3
sync_partial_ok:0
sync_partial_err:3
expired_keys:6
expired_stale_perc:0.00
expired_time_cap_reached_count:0
expire_cycle_cpu_milliseconds:21
evicted_keys:0
evicted_clients:0
total_eviction_exceeded_time:0
current_eviction_exceeded_time:0
keyspace_hits:232692
keyspace_misses:137713
pubsub_channels:1
pubsub_patterns:0
pubsubshard_channels:0
latest_fork_usec:624
total_forks:1
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0
total_active_defrag_time:0
current_active_defrag_time:0
tracking_total_keys:0
tracking_total_items:0
tracking_total_prefixes:0
unexpected_error_replies:0
total_error_replies:52
dump_payload_sanitizations:0
total_reads_processed:260421
total_writes_processed:279234
io_threaded_reads_processed:0
io_threaded_writes_processed:0
reply_buffer_shrinks:505
reply_buffer_expands:941
By wrinting that issue... I'm wondering if it's not because I point the exporter to a readonly endpoint...
It's still strange because the other metrics vary when my application hit the RW endpoint, so it's not really consistent in that case.
Can you run with the debug flag enabled to see what the difference might be?
Ok, I added the debug flag and the data get by "info" on the readonly endpoints is different.
I manualy made some checks, and in fact the RO endpoint in pointing to the slave replica. It's not pointing to the master as read only... It's a little bit confusing but that explains why hit & misses are wrong.
But... if redis expose shared data like the number of commands and their type on the replica, I don't understand why it's not consistent with the hit/miss values.
Do you think the correct approach in that case is to monitor the RW endpoint or the two endpoints ?
Yeah, monitor the RW instance. Or all of them, but separately (e.g. don't scrape a round-robin load-balanced endpoint)