unable to get metric http_requests: no metrics returned from custom metrics API
harshalk91 opened this issue · 11 comments
Hi,
I followed the instructions and deployed podinfo app and i am able to scale the app on http_requests. However, whenever i run my own app or just nginx, i am getting error as "unable to get metric http_requests: no metrics returned from custom metrics API". Deployment files are exactly the same, just changed port number, health_checks and labels.
[ec2-user@ip-192-168-100-253 nginx]$ kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
nginx-custom-hpa Deployment/nginx <unknown>/10 2 10 2 10m
podinfo Deployment/podinfo 888m/10 2 10 2 23m
[ec2-user@ip-192-168-100-253 nginx]$ kubectl describe hpa nginx-custom-hpa
Name: nginx-custom-hpa
Namespace: default
Labels: <none>
Annotations: <none>
CreationTimestamp: Mon, 19 Aug 2019 08:32:58 +0000
Reference: Deployment/nginx
Metrics: ( current / target )
"http_requests" on pods: <unknown> / 10
Min replicas: 2
Max replicas: 10
Deployment pods: 2 current / 0 desired
Conditions:
Type Status Reason Message
---- ------ ------ -------
AbleToScale True SucceededGetScale the HPA controller was able to get the target's current scale
ScalingActive False FailedGetPodsMetric the HPA was unable to compute the replica count: unable to get metric http_requests: no metrics returned from custom metrics API
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedComputeMetricsReplicas 8m18s (x12 over 11m) horizontal-pod-autoscaler failed to get object metric value: unable to get metric http_requests: no metrics returned from custom metrics API
Warning FailedGetPodsMetric 63s (x41 over 11m) horizontal-pod-autoscaler unable to get metric http_requests: no metrics returned from custom metrics API
Is my app running??
[ec2-user@ip-192-168-100-253 nginx]$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.100.0.1 <none> 443/TCP 21d
nginx-svc NodePort 10.100.195.60 <none> 9899:31199/TCP 13m
podinfo NodePort 10.100.112.143 <none> 9898:31198/TCP 24m
[ec2-user@ip-192-168-101-39 ~]$ curl -I http://10.100.195.60:9899
HTTP/1.1 200 OK
Server: nginx/1.17.3
[ec2-user@ip-192-168-100-253 ~]$ kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/*/http_requests" | jq .
{
"kind": "MetricValueList",
"apiVersion": "custom.metrics.k8s.io/v1beta1",
"metadata": {
"selfLink": "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/%2A/http_requests"
},
"items": [
{
"describedObject": {
"kind": "Pod",
"namespace": "default",
"name": "podinfo-7b8c9bc5c9-xgppt",
"apiVersion": "/v1"
},
"metricName": "http_requests",
"timestamp": "2019-08-19T08:48:38Z",
"value": "888m"
},
{
"describedObject": {
"kind": "Pod",
"namespace": "default",
"name": "podinfo-7b8c9bc5c9-zkkbd",
"apiVersion": "/v1"
},
"metricName": "http_requests",
"timestamp": "2019-08-19T08:48:38Z",
"value": "888m"
}
]
}
I would like to know why i am not able to see http_requests for nginx app?
@stefanprodan Any clue?
@stefanprodan What changes do i need to perform when i wish to run my apps under separate namespace?
Any update on this?, i'm running into same issue
Any update?
Guys, any update on this? Was any one able to achive this with nginx or any other service?
Please up vote this, we need help here
@krish512 Was wondering the same. How do we get the http_requests custom metrics from any pod we deploy? Is there something we need to insert/configure with the deployment or image itself? It only works with that podinfo deployment.
I guess the pod needs to expose this metrics and Custom Metrics Exporters can just capture these and make them available for HPA
Not sure if this is correct, can someone confirm if my understanding is correct
did anyone resolve this issue?
if you guys did can someone point me in the path as to the changes that need to be done to the existing pod to allow Prometheus to read data from it?
I was finally able to figure it out. Each webserver that needs monitoring(like the go example or apache or in our case nginx), has its own mechanism to expose metrics for prom to scrape. In the case of nginx, I had to enable the module nginx-module-vts, and then create a simple rule. Here is a link to my repo if you want to see.
https://github.com/ntman4real/nginxmets
Let me know if you have any questions, and I am now seeing how to do the same with springboot. More to come!
I was finally able to figure it out. Each webserver that needs monitoring(like the go example or apache or in our case nginx), has its own mechanism to expose metrics for prom to scrape. In the case of nginx, I had to enable the module nginx-module-vts, and then create a simple rule. Here is a link to my repo if you want to see.
Thanks i'm looking to expose metrics for Go & Node... but i'll check it out... Thanks!
Add a rule to custom-metrics-config-map.yaml:
- seriesQuery: '{__name__=~"^http_requests_.*",kubernetes_pod_name!="",kubernetes_namespace!=""}'
seriesFilters: []
resources:
overrides:
kubernetes_namespace:
resource: namespace
kubernetes_pod_name:
resource: pod
name:
matches: ^(.*)_(total)$
as: "${1}"
metricsQuery: sum(rate(<<.Series>>{<<.LabelMatchers>>}[1m])) by (<<.GroupBy>>)