Redis exporter not able to scrape metrics from AWS managed Elasticache (redis) cluster

Question

Redis exporter not able to scrape metrics from AWS managed Elasticache (redis) cluster

ashishthakur55525 opened this issue 5 months ago · 5 comments

ashishthakur55525 commented 5 months ago

Describe the problem
Redis exporter not able to scrape metrics from AWS managed Elasticache (redis) cluster. We are running redis exporter in AWS EKS cluster which should pull data from AWS managed(redis cluster). As per Github documentation (https://github.com/oliver006/redis_exporter?tab=readme-ov-file) added both redis URL as well exporter endpoint to prometheus scraping configuration (prometheus.yml). But I was able to get metrics (below) from exporters endpoint and from redis instance target, which looks like default metrics, nothing much.

**Exporter logs:**
time="2024-08-19T09:19:30Z" level=info msg="Redis Metrics Exporter v1.58.0    build date: 2024-03-06-10:50:36    sha1: 99ff138684a6726b5d589401b1b28f7bccdb2f97    Go: go1.22.1    GOOS: linux    GOARCH: amd64"
time="2024-08-19T09:19:30Z" level=info msg="Providing metrics at :9121/metrics"
time="2024-08-19T09:19:44Z" level=error msg="Couldn't connect to redis instance (redis://<myredisurl>.cache.amazonaws.com:6379)"
time="2024-08-19T09:19:47Z" level=error msg="Couldn't connect to redis instance (redis://<myredisurl>.cache.amazonaws.com:6379)"
time="2024-08-19T09:20:24Z" level=error msg="Couldn't connect to redis instance (redis://<myredisurl>.cache.amazonaws.com:6379)"
time="2024-08-19T09:20:35Z" level=error msg="Couldn't connect to redis instance (redis://<myredisurl>.cache.amazonaws.com:6379)"
time="2024-08-19T09:20:44Z" level=error msg="Couldn't connect to redis instance (redis://<myredisurl>.cache.amazonaws.com:6379)"
time="2024-08-19T09:20:47Z" level=error msg="Couldn't connect to redis instance (redis://<myredisurl>.cache.amazonaws.com:6379)"
time="2024-08-19T09:21:24Z" level=error msg="Couldn't connect to redis instance (redis://<myredisurl>.cache.amazonaws.com:6379)"

**Metrics coming in from redis exporter endpoint:**
up1{}
go_gc_duration_seconds{}
go_gc_duration_seconds_count{}
go_gc_duration_seconds_sum{}
go_goroutines{}
go_info{}
go_memstats_alloc_bytes{}
go_memstats_alloc_bytes_total{}
go_memstats_buck_hash_sys_bytes{}
go_memstats_frees_total{}
go_memstats_gc_sys_bytes{}
go_memstats_heap_alloc_bytes{}
go_memstats_heap_idle_bytes{}
go_memstats_heap_inuse_bytes{}
go_memstats_heap_objects{}
go_memstats_heap_released_bytes{}
go_memstats_heap_sys_bytes{}
go_memstats_last_gc_time_seconds{}
go_memstats_lookups_total{}
go_memstats_mallocs_total{}
go_memstats_mcache_inuse_bytes{}
go_memstats_mcache_sys_bytes{}
go_memstats_mspan_inuse_bytes{}
go_memstats_mspan_sys_bytes{}
go_memstats_next_gc_bytes{}
go_memstats_other_sys_bytes{}
go_memstats_stack_inuse_bytes{}
go_memstats_stack_sys_bytes{}
go_memstats_sys_bytes{}
go_threads{}
job_service:scrape_samples_post_metric_relabeling:count{}
job_service:scrape_samples_post_metric_relabeling:sum{}
job_service:scrape_samples_scraped:count{}
job_service:scrape_samples_scraped:sum{}
process_cpu_seconds_total{}
process_max_fds{}
process_open_fds{}
process_resident_memory_bytes{}
process_start_time_seconds{}
process_virtual_memory_bytes{}
process_virtual_memory_max_bytes{}
redis_exporter_build_info{}
redis_exporter_last_scrape_connect_time_seconds{}
redis_exporter_last_scrape_duration_seconds{}
redis_exporter_last_scrape_error{}
redis_exporter_scrape_duration_seconds_count{}
redis_exporter_scrape_duration_seconds_sum{}
redis_exporter_scrapes_total{}
redis_target_scrape_request_errors_total{}
redis_up{}
scrape_duration_seconds{}
scrape_samples_post_metric_relabeling{}
scrape_samples_scraped{}
scrape_series_added{}
up{}


**Metrics coming in from redis URL (instance) endpoint:**
up1{}
job_service:scrape_samples_post_metric_relabeling:count{}
job_service:scrape_samples_post_metric_relabeling:sum{}
job_service:scrape_samples_scraped:count{}
job_service:scrape_samples_scraped:sum{}
redis_exporter_build_info{}
redis_exporter_last_scrape_connect_time_seconds{}
redis_exporter_last_scrape_duration_seconds{}
redis_exporter_last_scrape_error{}
redis_exporter_scrape_duration_seconds_count{}
redis_exporter_scrape_duration_seconds_sum{}
redis_exporter_scrapes_total{}
redis_target_scrape_request_errors_total{}
redis_up{}
scrape_duration_seconds{}
scrape_samples_post_metric_relabeling{}
scrape_samples_scraped{}
scrape_series_added{}
up{}

What version of redis_exporter are you running?
Please run redis_exporter --version if you're not sure what version you're running.

$ ./redis_exporter --version
INFO[0000] Redis Metrics Exporter v1.58.0    build date: 2024-03-06-10:50:36    sha1: 99ff138684a6726b5d589401b1b28f7bccdb2f97    Go: go1.22.1    GOOS: linux    GOARCH: amd64
$

Redis version: 5.x

Running the exporter
What's the full command you're using to run the exporter? (please remove passwords and other sensitive data)
Please include details about env variables, command line parameters, your orchestration setup, etc.

Here are the env variable I used in exporter values.yaml file.
redisAddress: redis://<myredisURL>.cache.amazonaws.com:6379

auth:
  enabled: true
  redisPassword: "<auth token to redis>"

Configuration form prometheus.yaml

   ## to scrape redis URL 
  - job_name: redis_exporter_targets
    static_configs:
      - targets:
        - redis://app-stage-redis.nh8wos.use2.cache.amazonaws.com:6379
    metrics_path: /scrape
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: prometheus-redis-exporter:9121

  ## config for scraping the exporter itself
  - job_name: 'redis_exporter'
     static_configs:
      - targets:
        - prometheus-redis-exporter:9121
     metrics_path: /metrics

Screenshots
If applicable, add screenshots to help explain your question.
this is screenshot from prometheus targets.

Not able to upload image, not sure. But these are two endpoints discovered in in prometheus targets.

1st endpoint
endpoint: http://prometheus-redis-exporter:9121/metrics
discovered labels
address="prometheus-redis-exporter:9121"
metrics_path="/metrics"
scheme="http"
scrape_interval="1m"
scrape_timeout="10s"
job="redis_exporter"

2nd endpoint
[http://prometheus-redis-exporter:9121/scrape?target="redis://myredisURL.cache.amazonaws.com:6379](http://prometheus-redis-exporter:9121/scrape?target=%22redis://myredisURL.cache.amazonaws.com:6379)"
discovered labels
address="redis://app-stage-redis.nh8wos.use2.cache.amazonaws.com:6379"
metrics_path="/scrape"
scheme="http"
scrape_interval="1m"
scrape_timeout="10s"
job="redis_exporter_targets"

Also how to get IP address in target instead of hostname(prometheus-redis-exporter).

Additional context

What we need to do to make redis (AWS managed) scraped from EKS cluster running exporter, how to get custom metrics published, adding targets to promtheus.yaml how its gonna scrape it, can we use servicemonitor instead ro scrape it, are there two endpoints /scrape & /metrics?

oliver006 commented 5 months ago

Not yet.

Answer 1 · 2024-08-19T11:46:18.000Z

Debug mode output:

$ ./redis_exporter --debug
INFO[0000] Redis Metrics Exporter v1.58.0    build date: 2024-03-06-10:50:36    sha1: 99ff138684a6726b5d589401b1b28f7bccdb2f97    Go: go1.22.1    GOOS: linux    GOARCH: amd64
DEBU[0000] Enabling debug output
DEBU[0000] NewRedisExporter options: exporter.Options{User:"", Password:"<password>", Namespace:"redis", PasswordMap:map[string]string{}, ConfigCommandName:"CONFIG", CheckKeys:"", CheckSingleKeys:"", CheckStreams:"", CheckSingleStreams:"", CheckKeysBatchSize:1000, CheckKeyGroups:"", MaxDistinctKeyGroups:100, CountKeys:"", LuaScript:map[string][]uint8(nil), ClientCertFile:"", ClientKeyFile:"", CaCertFile:"", InclConfigMetrics:false, DisableExportingKeyValues:false, RedactConfigMetrics:true, InclSystemMetrics:false, SkipTLSVerification:false, SetClientName:true, IsTile38:false, IsCluster:false, ExportClientList:false, ExportClientsInclPort:false, ConnectionTimeouts:15000000000, MetricsPath:"/metrics", RedisMetricsOnly:false, PingOnConnect:false, RedisPwdFile:"", Registry:(*prometheus.Registry)(0xd1b980), BuildInfo:exporter.BuildInfo{Version:"v1.58.0", CommitSha:"99ff138684a6726b5d589401b1b28f7bccdb2f97", Date:"2024-03-06-10:50:36"}}
DEBU[0000] parseKeyArg(): Got empty key arguments, parsing skipped
DEBU[0000] keys: []exporter.dbKeyPair(nil)
DEBU[0000] parseKeyArg(): Got empty key arguments, parsing skipped
DEBU[0000] singleKeys: []exporter.dbKeyPair(nil)
DEBU[0000] parseKeyArg(): Got empty key arguments, parsing skipped
DEBU[0000] streams: []exporter.dbKeyPair(nil)
DEBU[0000] parseKeyArg(): Got empty key arguments, parsing skipped
DEBU[0000] singleStreams: []exporter.dbKeyPair(nil)
DEBU[0000] parseKeyArg(): Got empty key arguments, parsing skipped
DEBU[0000] countKeys: []exporter.dbKeyPair(nil)
INFO[0000] Providing metrics at :9121/metrics
DEBU[0000] Configured redis addr: "redis://<myredisURL>.cache.amazonaws.com:6379"
FATA[0000] Server error: listen tcp :9121: bind: address already in use

Answer 2 · 2024-08-28T06:22:07.000Z

@oliver006 any chance were you able to look into this issue?

Answer 3 · 2024-08-29T06:38:17.000Z

Not yet.

sure, please let me know if you have any update?
Also is my configuration correct?

Answer 4 · 2024-09-05T06:41:33.000Z

@oliver006 i think you can close this out, i was able to scrape AWS managed redis using servicemonitor deployment, and enabling TLS.