Prometheus integration - scraping API server metrics

[kamilzzz]

Is it possible to scrape AKS API server metrics using https://docs.microsoft.com/en-us/azure/azure-monitor/containers/container-insights-prometheus-integration?

As far as I know, to get /metrics from API server authentication is required (bearer token) and I cannot see how this can be set in the monitoring agent config file https://docs.microsoft.com/en-us/azure/azure-monitor/containers/container-insights-prometheus-integration#prometheus-scraping-settings.

For standard Prometheus deployment this can be configured via bearer_token_file setting.

[github-actions]

This issue is stale because it has been open 7 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[vishiy]

@kamilzzz - yes, you can scrape API server. Use the URI - https://$KUBERNETES_SERVICE_HOST:$KUBERNETES_SERVICE_PORT/metrics

See below for the relevant section in the configmap as sample. Our agent pod has access to scrape api server --

prometheus-data-collection-settings: |-
    # Custom Prometheus metrics data collection settings
    [prometheus_data_collection_settings.cluster]
        # Cluster level scrape endpoint(s). These metrics will be scraped from agent's Replicaset (singleton)
        # Any errors related to prometheus scraping can be found in the KubeMonAgentEvents table in the Log Analytics workspace that the cluster is sending data to.

        #Interval specifying how often to scrape for metrics. This is duration of time and can be specified for supporting settings by combining an integer value and time unit as a string value. Valid time units are ns, us (or µs), ms, s, m, h.
        interval = "1m"

        ## Uncomment the following settings with valid string arrays for prometheus scraping
        #fieldpass = ["etcd_object_counts"]

        #fielddrop = ["metric_to_drop"]

        # An array of urls to scrape metrics from.
        urls = ["https://$KUBERNETES_SERVICE_HOST:$KUBERNETES_SERVICE_PORT/metrics"]

        # An array of Kubernetes services to scrape metrics from.
        # kubernetes_services = ["http://my-service-dns.my-namespace:9102/metrics"]

        # When monitor_kubernetes_pods = true, replicaset will scrape Kubernetes pods for the following prometheus annotations:
        # - prometheus.io/scrape: Enable scraping for this pod
        # - prometheus.io/scheme: If the metrics endpoint is secured then you will need to
        #     set this to `https` & most likely set the tls config.
        # - prometheus.io/path: If the metrics path is not /metrics, define it with this annotation.
        # - prometheus.io/port: If port is not 9102 use this annotation
        monitor_kubernetes_pods = false

        ## Restricts Kubernetes monitoring to namespaces for pods that have annotations set and are scraped using the monitor_kubernetes_pods setting.
        ## This will take effect when monitor_kubernetes_pods is set to true
        ##   ex: monitor_kubernetes_pods_namespaces = ["default1", "default2", "default3"]
        # monitor_kubernetes_pods_namespaces = ["default1"]

[vishiy]

also please keep an eye on the volume/cost as this can be a lot of metrics. As needed use 'fieldpass' to filter-in only metrics that you need.