oliver006/redis_exporter

Add option to only run check and counts on replicas

jdheyburn opened this issue · 4 comments

jdheyburn commented

Describe the problem

redis-exporter has these arguments available for exporting the length of particular keys:

check-keys
check-single-keys
check-streams
check-single-streams
count-keys

It's described in the docs that these commands may not perform well on large databases, and that it can affect production environments.

My questions:

  1. Would it make sense to have an option that only allows these to execute on replicas?
  2. Can these be enhanced to gather the memory consumed by the keys too via MEMORY USAGE?
  3. Can both these problems be solved by using a custom lua script defined in REDIS_EXPORTER_SCRIPT?

What version of redis_exporter are you running?
Please run redis_exporter --version if you're not sure what version you're running.
[x] 1.6.2

Running the exporter
N/A

Screenshots
N/A

Additional context
N/A

oliver006 commented

Good questions!

  1. what's your deployment setup? A redis_exporter sidecar with each redis instance? Becuase in that case you can just configure them differently and only run check-keys on the replica instances.

  2. I have not looked into using MEMORY USAGE and no idea re: performance comparision but worth testing

  3. You could certainly replicate the check-keys functionality with a lua script and see if e.g. MEMORY USAGE performs better. The lua script option works well when you have defined inputs and need to return only a single float so this could be a good option.

I don't have time to investigate option (2) but that could be an easy win if that performs a lot better than GET'ing each key. (the results wouldn't be equivalent cause MEMORY USAGE reports key + val size but it's still a god approximation and can be useful. I'd put that functionality on a cmd line flag and keep the current functionality by default for backwards compatibility)

jdheyburn commented

Thanks for taking a look @oliver006!

what's your deployment setup? A redis_exporter sidecar with each redis instance? Becuase in that case you can just configure them differently and only run check-keys on the replica instances.

We use a fork of the DandyDeveloper redis-ha helm-chart. Each pod in the statefulset has the same configuration, one of the pods will be a master and the rest would be replicas.

I have not looked into using MEMORY USAGE and no idea re: performance comparision but worth testing

This frequently shows up in the slow log on workloads today where we execute it on our replicas (for troubleshooting high memory issues) - so I would avoid running it on a master

oliver006 commented

Interesting - so the master vs replica decision is made on the fly and not pre-determined?
I guess we could add a flag that allows for skipping the check-keys metrics if role==master, defaulting to false.

jdheyburn commented

@oliver006

Interesting - so the master vs replica decision is made on the fly and not pre-determined?

Correct. On StatefulSet provision, the first ordinal pod would be the master (as there is nothing to replicate from). Subsequent pods would be instructed to replicate from the master.

However, we have AZ affinity in our set up, this ordinal pod may not be running in the desired AZ. For that we have a separate process to invoke a failover to the pod running in the desired AZ.

I guess we could add a flag that allows for skipping the check-keys metrics if role==master, defaulting to false.

That would be ideal, I won't have time to pick it up until later this year though.

  • 👍1