prometheus-community/helm-charts

[kube-prometheus-stack] windows-exporter unable to collect IIS Metrics

1d9akash opened this issue · 0 comments

1d9akash commented

The windows-exporter in the kube-prometheus-stack is unable to collect IIS metrics. When I check the logs of the Windows Exporter pod, it shows repeated "counter not found" errors.

Logs:

kubectl logs -f pod/kube-prometheus-stack-prometheus-windows-exporter-vvtfq -n monitoring

Defaulted container "windows-exporter" out of: windows-exporter, configure-firewall (init)
ts=2024-05-17T11:39:07.365Z caller=config.go:63 level=info msg="Loading configuration file: C:\\C\\081c8ad10541cc196120210773c6068661b3a863df957d188521a32855b4951b/config.yml"
ts=2024-05-17T11:39:07.391Z caller=service.go:93 level=warn collector=service msg="No where-clause specified for service collector. This will generate a very large number of metrics!"
ts=2024-05-17T11:39:07.393Z caller=iis.go:51 level=warn collector=iis msg="Couldn't open registry to determine IIS version" err="The system cannot find the file specified."
ts=2024-05-17T11:39:08.288Z caller=exporter.go:165 level=info msg="Running as NT AUTHORITY\\SYSTEM"
ts=2024-05-17T11:39:08.288Z caller=exporter.go:172 level=info msg="Enabled collectors: memory, iis, net, netframework_clr, cs, system, cpu, service, container, os, logical_disk, process, netframework_clrexceptions"
ts=2024-05-17T11:39:08.288Z caller=exporter.go:225 level=info msg="Starting windows_exporter" version="(version=0.25.1, branch=heads/tags/v0.25.1, revision=f70fa009de541dc99ed210aa7e67c9550133ef02)"
ts=2024-05-17T11:39:08.288Z caller=exporter.go:226 level=info msg="Build context" build_context="(go=go1.21.5, platform=windows/amd64, user=runneradmin@fv-az844-399, date=20240116-17:56:36, tags=unknown)"
ts=2024-05-17T11:39:08.290Z caller=tls_config.go:313 level=info msg="Listening on" address=[::]:9182
ts=2024-05-17T11:39:08.290Z caller=tls_config.go:316 level=info msg="TLS is disabled." http2=false address=[::]:9182
ts=2024-05-17T11:39:25.902Z caller=iis.go:919 level=error collector=iis msg="failed collecting iis metrics" desc=null err="counter not found"
ts=2024-05-17T11:39:25.902Z caller=prometheus.go:186 level=error msg="collector iis failed after 0.001452s" err="counter not found"
ts=2024-05-17T11:39:55.884Z caller=iis.go:919 level=error collector=iis msg="failed collecting iis metrics" desc=null err="counter not found"
ts=2024-05-17T11:39:55.885Z caller=prometheus.go:186 level=error msg="collector iis failed after 0.000000s" err="counter not found"
ts=2024-05-17T11:40:25.889Z caller=iis.go:919 level=error collector=iis msg="failed collecting iis metrics" desc=null err="counter not found"
ts=2024-05-17T11:40:25.891Z caller=prometheus.go:186 level=error msg="collector iis failed after 0.000543s" err="counter not found"
ts=2024-05-17T11:40:55.886Z caller=iis.go:919 level=error collector=iis msg="failed collecting iis metrics" desc=null err="counter not found"
ts=2024-05-17T11:40:55.887Z caller=prometheus.go:186 level=error msg="collector iis failed after 0.000000s" err="counter not found"
ts=2024-05-17T11:41:25.890Z caller=iis.go:919 level=error collector=iis msg="failed collecting iis metrics" desc=null err="counter not found"
ts=2024-05-17T11:41:25.891Z caller=prometheus.go:186 level=error msg="collector iis failed after 0.001175s" err="counter not found"
ts=2024-05-17T11:41:55.889Z caller=iis.go:919 level=error collector=iis msg="failed collecting iis metrics" desc=null err="counter not found"
ts=2024-05-17T11:41:55.890Z caller=prometheus.go:186 level=error msg="collector iis failed after 0.001446s" err="counter not found"
ts=2024-05-17T11:42:25.888Z caller=iis.go:919 level=error collector=iis msg="failed collecting iis metrics" desc=null err="counter not found"
ts=2024-05-17T11:42:25.888Z caller=prometheus.go:186 level=error msg="collector iis failed after 0.000000s" err="counter not found"
ts=2024-05-17T11:42:55.890Z caller=iis.go:919 level=error collector=iis msg="failed collecting iis metrics" desc=null err="counter not found"
ts=2024-05-17T11:42:55.890Z caller=prometheus.go:186 level=error msg="collector iis failed after 0.001637s" err="counter not found"
time="2024-05-17T11:42:55Z" level=error msg="failed to waitForNotification: callbackNumber does not exist in callbackMap" callbackNumber=71
time="2024-05-17T11:42:55Z" level=error msg="failed to waitForNotification: callbackNumber does not exist in callbackMap" callbackNumber=72

What's your helm version?

3.13.3

What's your kubectl version?

1.27.2

Which chart?

kube-prometheus-stack

What's the chart version?

58.5.1

What happened?

The windows-exporter is unable to collect IIS metrics. The logs show repeated "counter not found" errors.

What you expected to happen?

The windows-exporter should be able to collect IIS metrics without any errors.

How to reproduce it?

  1. Enable the windows-exporter in the kube-prometheus-stack chart.
  2. Under the configuration of prometheus-windows-exporter, add 'iis' with default config and check the logs of the windows-exporter pod.

Enter the changed values of values.yaml?

windowsMonitoring:
prometheus-windows-exporter:
  prometheus:
    monitor:
      enabled: true
      jobLabel: jobLabel
  releaseLabel: true
  podLabels:
    jobLabel: windows-exporter
  config: |-
    collectors:
      enabled: 'memory,container,iis,cs,netframework_clrexceptions,os,system,logical_disk,net,process,cpu,netframework_clr,service'

Enter the command that you execute and failing/misfunctioning.

kubectl logs -f pod/kube-prometheus-stack-prometheus-windows-exporter-vvtfq -n monitoring

Anything else we need to know?

The necessary performance counters for IIS are available on the system where the windows-exporter is running, as confirmed by running Get-Counter -ListSet * in PowerShell.

Here it is for Webservice:

CounterSetName     : Web Service
MachineName        : .
CounterSetType     : MultiInstance
Description        : The Web Service object includes counters specific to the World Wide Web Publishing Service.
Paths              : {\Web Service(*)\Total Bytes Sent, \Web Service(*)\Bytes Sent/sec, \Web Service(*)\Total Bytes Received, \Web Service(*)\Bytes Received/sec...}
PathsWithInstances : {\Web Service(_Total)\Total Bytes Sent, \Web Service(HTTPTransfer)\Total Bytes Sent, \Web Service(MOBILITY_ARR)\Total Bytes Sent, \Web Service(BC_ARR_CLUSTER)\Total Bytes Sent...}
Counter            : {\Web Service(*)\Total Bytes Sent, \Web Service(*)\Bytes Sent/sec, \Web Service(*)\Total Bytes Received, \Web Service(*)\Bytes Received/sec...}