ricoberger/script_exporter

gaps between the metrics

bobykus31 opened this issue · 2 comments

bobykus31 commented

Since we started using script_exporter, we found a weird issue that makes us uncomfortable. We use script_exporter to execute some long-running scripts like smartctl metrics. Even though we have a reasonable cache duration set up, we still have gaps in between the metrics, that correspond to the time of script execution or close. The script-exporter.yml looks like

scripts:
...
- name: node_smartmon
  command: "/usr/lib/prometheus/custom_metric_node_smartmon.sh"
  cacheDuration: 1200s
  timeout:
    max_timeout: 600
    enforced: false

The result is

image

Is there any good way to get rid of this gaps?

ricoberger commented

Hi @bobykus31, the problem might be the max_timeout of 600 seconds, which is larger as the default staleness of 5 minutes. So when the script is executed and the scrape takes more the 5 minutes, it would result in a small gap until the execution of the scrape is finished. Can you check if the gaps are corresponding with the script_duration_seconds metric when it is larger than 5 minutes, please?

ricoberger commented

Hi @bobykus31, since there is no activity for some time, I will close the issue for now. If this is still a problem please let me know and we can reopen the issue.