custom autoscaler not working
manishrajkarnikar opened this issue · 7 comments
✗ kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1/namespaces/default/services/sample-metrics-app/http_requests | jq
{
"kind": "MetricValueList",
"apiVersion": "custom.metrics.k8s.io/v1beta1",
"metadata": {
"selfLink": "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/services/sample-metrics-app/http_requests"
},
"items": [
{
"describedObject": {
"kind": "Service",
"name": "sample-metrics-app",
"apiVersion": "/__internal"
},
"metricName": "http_requests",
"timestamp": "2017-11-29T21:31:13Z",
"value": "539m"
}
]
}
Note that http_requests has value of "539m". It should be a number and not memory. In my case I was expecting 120 (that was my transaction per second i had set via my load test tool). Due to this issue hpa does not scale the replicaset appropriately. Prometheus metrics was verified to show right value.
I1129 21:27:07.806751 1 api.go:74] GET http://sample-metrics-prom.default.svc:9090/api/v1/query?query=sum%28rate%28http_requests_total%7Bservice%3D%22sample-metrics-app%22%2Cnamespace%3D%22default%22%7D%5B
1m%5D%29%29+by+%28service%29&time=1511990827.805 200 OK
I1129 21:27:07.806847 1 api.go:93] Response Body: {"status":"success","data":{"resultType":"vector","result":[{"metric":{"service":"sample-metrics-app"},"value":[1511990827.805,"30.54"]}]}}
This is what describing the hpa shows.
➜ kubectl describe hpa sample-metrics-app-hpa
Name: sample-metrics-app-hpa
Namespace: default
Labels: <none>
Annotations: kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"autoscaling/v2beta1","kind":"HorizontalPodAutoscaler","metadata":{"annotations":{},"name":"sample-metrics-app-hpa","namespace":"default"...
CreationTimestamp: Wed, 29 Nov 2017 21:29:58 +0000
Reference: Deployment/sample-metrics-app
Metrics: ( current / target )
"http_requests" on Service/sample-metrics-app: 125300m / 100
Min replicas: 1
Max replicas: 10
Conditions:
Type Status Reason Message
---- ------ ------ -------
AbleToScale True ReadyForNewScale the last scale time was sufficiently old as to warrant a new scale
ScalingActive True ValidMetricFound the HPA was able to succesfully calculate a replica count from Service metric http_requests
ScalingLimited True TooManyReplicas the desired replica count was more than the maximum replica count
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedGetObjectMetric 8m (x9 over 2h) horizontal-pod-autoscaler unable to get metric http_requests: Service on default sample-metrics-app/unable to fetch metrics from API: the server could not find the requested resource (get services.custom.metrics.k8s.io sample-metrics-app)
Warning FailedComputeMetricsReplicas 8m (x9 over 2h) horizontal-pod-autoscaler failed to get object metric value: unable to get metric http_requests: Service on default sample-metrics-app/unable to fetch metrics from API: the server could not find the requested resource (get services.custom.metrics.k8s.io sample-metrics-app)
i am running into the same problem, have u solved it?
I've got the same issue
I got same issue as well:
[root@kvm-014724 autoscaling]# kubectl describe hpa
Name: sample-metrics-app-hpa
Namespace: default
Labels: <none>
Annotations: <none>
CreationTimestamp: Wed, 28 Feb 2018 12:07:33 +0800
Reference: Deployment/sample-metrics-app
Metrics: ( current / target )
"http_requests" on Service/sample-metrics-app: <unknown> / 100
Min replicas: 2
Max replicas: 10
Conditions:
Type Status Reason Message
---- ------ ------ -------
AbleToScale True SucceededGetScale the HPA controller was able to get the target's current scale
ScalingActive False FailedGetObjectMetric the HPA was unable to compute the replica count: unable to get metric http_requests: Service on default sample-metrics-app/unable to fetch metrics from API: the server could not find the requested resource (get services.custom.metrics.k8s.io sample-metrics-app)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedGetObjectMetric 7m (x101 over 57m) horizontal-pod-autoscaler unable to get metric http_requests: Service on default sample-metrics-app/unable to fetch metrics from API: the server could not find the requested resource (get services.custom.metrics.k8s.io sample-metrics-app)
Warning FailedComputeMetricsReplicas 2m (x111 over 57m) horizontal-pod-autoscaler failed to get object metric value: unable to get metric http_requests: Service on default sample-metrics-app/unable to fetch metrics from API: the server could not find the requested resource (get services.custom.metrics.k8s.io sample-metrics-app)
@luxas could you help double check it, thanks!
I noticed that the sample server outputs metric http_requests_total (https://github.com/luxas/kubeadm-workshop/blob/master/images/autoscaling/server.js#L12), while the HPA monitors http_requests (https://github.com/luxas/kubeadm-workshop/blob/master/demos/monitoring/sample-metrics-app.yaml#L95) Not sure if that’s intentional. I’ve tried both metric names in my HPA and no luck
@zihaoyu @huxiaoliang I have found a work around. As @zihaoyu mentioned the server app is giving out http_requests_total while the prometheus adapter is looking out for http_request(figuring out why ) , So what we did is that we modified the application (https://github.com/luxas/kubeadm-workshop/blob/master/images/autoscaling/server.js ) to use http_request and redeployed the container and autoscaling started working
@luxas Can you tell how the http_requests in hpa is getting resolved? I have looked at the default rules in DirectXMan12/k8s-prometheus-adapter, there seems like no default rule for this.