Long Time to Return Metrics

[dfcarver]

When we first implemented this it was taking less than 30 seconds:
May 29 03:48:20 rmq-01 rabbitmq_exporter: time="2022-05-29T03:48:20-07:00" level=info msg="Metrics updated" duration=12.704870398s
May 29 03:48:50 rmq-01 rabbitmq_exporter: time="2022-05-29T03:48:50-07:00" level=info msg="Metrics updated" duration=13.002259512s
May 29 03:49:20 rmq-01 rabbitmq_exporter: time="2022-05-29T03:49:20-07:00" level=info msg="Metrics updated" duration=12.941314835s
May 29 03:49:50 rmq-01 rabbitmq_exporter: time="2022-05-29T03:49:50-07:00" level=info msg="Metrics updated" duration=12.897284942s
​It's basically slowly getting worse over time:
Jun 12 03:34:32 rmq-01 rabbitmq_exporter: time="2022-06-12T03:34:32-07:00" level=info msg="Metrics updated" duration=24.344881875s
Jun 12 03:35:01 rmq-01 rabbitmq_exporter: time="2022-06-12T03:35:01-07:00" level=info msg="Metrics updated" duration=23.783680561s
Jun 12 03:35:31 rmq-01 rabbitmq_exporter: time="2022-06-12T03:35:31-07:00" level=info msg="Metrics updated" duration=23.718060753s
Jun 12 03:36:01 rmq-01 rabbitmq_exporter: time="2022-06-12T03:36:01-07:00" level=info msg="Metrics updated" duration=24.139322817s
​And now it's taking 2, 3, sometimes 4 minutes to update the metrics:
Jul 1 12:03:38 rmq-01 rabbitmq_exporter: time="2022-07-01T12:03:38-07:00" level=info msg="Metrics updated" duration=2m0.793239306s
Jul 1 12:10:23 rmq-01 rabbitmq_exporter: time="2022-07-01T12:10:23-07:00" level=info msg="Metrics updated" duration=2m3.832563396s
Jul 1 12:14:28 rmq-01 rabbitmq_exporter: time="2022-07-01T12:14:28-07:00" level=info msg="Metrics updated" duration=2m8.298392559s
Jul 1 12:18:55 rmq-01 rabbitmq_exporter: time="2022-07-01T12:18:55-07:00" level=info msg="Metrics updated" duration=2m34.94214455s
​I've tried restarting the rabbitmq_exporter process but that does not seem like it has helped.

Any ideas what's going on here? Would reducing the number of metrics help?

[kbudde]

Hi,

it's most probably a slow response time of the rabbitMQ itself.
Which version of the rabbitMQ exporter is in use?
The capabilites "bert,nosort" are quite useful to speed up the scraping.

I suspect that the queue endpoint of the API is the slowest.
How long does it take to call http://rabbitmq:15672/api/queues?sort=

[dfcarver]

Hi, Kris, We are running RC15, the API is taking around 60 second and our configuration is below: { "rabbit_url": "https://127.0.0.1:15672", "rabbit_user": "opsmonitor", "rabbit_pass": "xxxxxxx", "publish_port": "9419", "publish_addr": "", "output_format": "TTY", "ca_file": "ca.pem", "cert_file": "client-cert.pem", "key_file": "client-key.pem", "insecure_skip_verify": true, "exlude_metrics": [], "include_exchanges": ".*", "skip_exchanges": "^$", "include_queues": ".*", "skip_queues": "^$", "skip_vhost": "^$", "include_vhost": ".*", "rabbit_capabilities": "no_sort,bert", "aliveness_vhost": "/", "enabled_exporters": [ "exchange", "node", "overview", "queue", "aliveness" ], "timeout": 240, "max_queues": 0 } Kindly advise.

…

On Jul 2, 2022, at 2:46 PM, Kris Budde ***@***.***> wrote: Hi, it's most probably a slow response time of the rabbitMQ itself. Which version of the rabbitMQ exporter is in use? The capabilites "bert,nosort" are quite useful to speed up the scraping. I suspect that the queue endpoint of the API is the slowest. How long does it take to call http://rabbitmq:15672/api/queues?sort= <http://rabbitmq:15672/api/queues?sort=> — Reply to this email directly, view it on GitHub <#287 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AC5SMWNC62LHJXDTRJERNETVSCE73ANCNFSM52NYHR6A>. You are receiving this because you authored the thread.

[kbudde]

@dfcarver this is already a long time. I'm not sure if I can do anything.
The duration should only increase with some relation to your rabbitmq usage (number of queues, etc.).

Can you please check the metric rabbitmq_module_scrape_duration_seconds.
Maybe it's the aliveness endpoint. This might also cause some delay.