giantswarm/prometheus

Prometheus failed to list ressources

Closed this issue · 5 comments

trahloff commented

Seems like the Prometheus service can't access k8s ressources:

2017-09-05T05:55:25.325516476Z time="2017-09-05T05:55:25Z" level=error msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:227: Failed to list *v1.Service: Forbidden: "/api/v1/services?resourceVersion=0" (get services)" component="kube_client_runtime" source="kubernetes.go:75" 
2017-09-05T05:55:25.544671315Z time="2017-09-05T05:55:25Z" level=error msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:179: Failed to list *v1.Endpoints: Forbidden: "/api/v1/endpoints?resourceVersion=0" (get endpoints)" component="kube_client_runtime" source="kubernetes.go:75" 
2017-09-05T05:55:25.645500647Z time="2017-09-05T05:55:25Z" level=error msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:181: Failed to list *v1.Pod: Forbidden: "/api/v1/pods?resourceVersion=0" (get pods)" component="kube_client_runtime" source="kubernetes.go:75" 
2017-09-05T05:55:25.703932970Z time="2017-09-05T05:55:25Z" level=error msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:207: Failed to list *v1.Pod: Forbidden: "/api/v1/pods?resourceVersion=0" (get pods)" component="kube_client_runtime" source="kubernetes.go:75" 
2017-09-05T05:55:25.794057778Z time="2017-09-05T05:55:25Z" level=error msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:245: Failed to list *v1.Node: Forbidden: "/api/v1/nodes?resourceVersion=0" (get nodes)" component="kube_client_runtime" source="kubernetes.go:75" 
2017-09-05T05:55:25.794077568Z time="2017-09-05T05:55:25Z" level=error msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:180: Failed to list *v1.Service: Forbidden: "/api/v1/services?resourceVersion=0" (get services)" component="kube_client_runtime" source="kubernetes.go:75" 
2017-09-05T05:55:26.332010158Z time="2017-09-05T05:55:26Z" level=error msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:227: Failed to list *v1.Service: Forbidden: "/api/v1/services?resourceVersion=0" (get services)" component="kube_client_runtime" source="kubernetes.go:75" 
2017-09-05T05:55:26.547030910Z time="2017-09-05T05:55:26Z" level=error msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:179: Failed to list *v1.Endpoints: Forbidden: "/api/v1/endpoints?resourceVersion=0" (get endpoints)" component="kube_client_runtime" source="kubernetes.go:75" 
2017-09-05T05:55:26.648655345Z time="2017-09-05T05:55:26Z" level=error msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:181: Failed to list *v1.Pod: Forbidden: "/api/v1/pods?resourceVersion=0" (get pods)" component="kube_client_runtime" source="kubernetes.go:75" 
2017-09-05T05:55:26.706663533Z time="2017-09-05T05:55:26Z" level=error msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:207: Failed to list *v1.Pod: Forbidden: "/api/v1/pods?resourceVersion=0" (get pods)" component="kube_client_runtime" source="kubernetes.go:75" 
2017-09-05T05:55:26.800311617Z time="2017-09-05T05:55:26Z" level=error msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:180: Failed to list *v1.Service: Forbidden: "/api/v1/services?resourceVersion=0" (get services)" component="kube_client_runtime" source="kubernetes.go:75" 
2017-09-05T05:55:26.800331822Z time="2017-09-05T05:55:26Z" level=error msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:245: Failed to list *v1.Node: Forbidden: "/api/v1/nodes?resourceVersion=0" (get nodes)" component="kube_client_runtime" source="kubernetes.go:75" 
2017-09-05T05:55:27.335507760Z time="2017-09-05T05:55:27Z" level=error msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:227: Failed to list *v1.Service: Forbidden: "/api/v1/services?resourceVersion=0" (get services)" component="kube_client_runtime" source="kubernetes.go:75" 
2017-09-05T05:55:27.549759362Z time="2017-09-05T05:55:27Z" level=error msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:179: Failed to list *v1.Endpoints: Forbidden: "/api/v1/endpoints?resourceVersion=0" (get endpoints)" component="kube_client_runtime" source="kubernetes.go:75" 
2017-09-05T05:55:27.651208222Z time="2017-09-05T05:55:27Z" level=error msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:181: Failed to list *v1.Pod: Forbidden: "/api/v1/pods?resourceVersion=0" (get pods)" component="kube_client_runtime" source="kubernetes.go:75" 
2017-09-05T05:55:27.709451754Z time="2017-09-05T05:55:27Z" level=error msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:207: Failed to list *v1.Pod: Forbidden: "/api/v1/pods?resourceVersion=0" (get pods)" component="kube_client_runtime" source="kubernetes.go:75" 
2017-09-05T05:55:27.802812402Z time="2017-09-05T05:55:27Z" level=error msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:180: Failed to list *v1.Service: Forbidden: "/api/v1/services?resourceVersion=0" (get services)" component="kube_client_runtime" source="kubernetes.go:75" 
2017-09-05T05:55:27.802836038Z time="2017-09-05T05:55:27Z" level=error msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:245: Failed to list *v1.Node: Forbidden: "/api/v1/nodes?resourceVersion=0" (get nodes)" component="kube_client_runtime" source="kubernetes.go:75" 
2017-09-05T05:55:28.338185252Z time="2017-09-05T05:55:28Z" level=error msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:227: Failed to list *v1.Service: Forbidden: "/api/v1/services?resourceVersion=0" (get services)" component="kube_client_runtime" source="kubernetes.go:75" 
2017-09-05T05:55:28.553583658Z time="2017-09-05T05:55:28Z" level=error msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:179: Failed to list *v1.Endpoints: Forbidden: "/api/v1/endpoints?resourceVersion=0" (get endpoints)" component="kube_client_runtime" source="kubernetes.go:75" 
2017-09-05T05:55:28.655636436Z time="2017-09-05T05:55:28Z" level=error msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:181: Failed to list *v1.Pod: Forbidden: "/api/v1/pods?resourceVersion=0" (get pods)" component="kube_client_runtime" source="kubernetes.go:75" 
2017-09-05T05:55:28.711802173Z time="2017-09-05T05:55:28Z" level=error msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:207: Failed to list *v1.Pod: Forbidden: "/api/v1/pods?resourceVersion=0" (get pods)" component="kube_client_runtime" source="kubernetes.go:75" 
2017-09-05T05:55:28.805598875Z time="2017-09-05T05:55:28Z" level=error msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:180: Failed to list *v1.Service: Forbidden: "/api/v1/services?resourceVersion=0" (get services)" component="kube_client_runtime" source="kubernetes.go:75" 
2017-09-05T05:55:28.805619270Z time="2017-09-05T05:55:28Z" level=error msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:245: Failed to list *v1.Node: Forbidden: "/api/v1/nodes?resourceVersion=0" (get nodes)" component="kube_client_runtime" source="kubernetes.go:75" 
2017-09-05T05:55:29.342557645Z time="2017-09-05T05:55:29Z" level=error msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:227: Failed to list *v1.Service: Forbidden: "/api/v1/services?resourceVersion=0" (get services)" component="kube_client_runtime" source="kubernetes.go:75" 
2017-09-05T05:55:29.555876317Z time="2017-09-05T05:55:29Z" level=error msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:179: Failed to list *v1.Endpoints: Forbidden: "/api/v1/endpoints?resourceVersion=0" (get endpoints)" component="kube_client_runtime" source="kubernetes.go:75" 
2017-09-05T05:55:29.658413209Z time="2017-09-05T05:55:29Z" level=error msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:181: Failed to list *v1.Pod: Forbidden: "/api/v1/pods?resourceVersion=0" (get pods)" component="kube_client_runtime" source="kubernetes.go:75" 
2017-09-05T05:55:29.716340932Z time="2017-09-05T05:55:29Z" level=error msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:207: Failed to list *v1.Pod: Forbidden: "/api/v1/pods?resourceVersion=0" (get pods)" component="kube_client_runtime" source="kubernetes.go:75" 
2017-09-05T05:55:29.808308737Z time="2017-09-05T05:55:29Z" level=error msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:245: Failed to list *v1.Node: Forbidden: "/api/v1/nodes?resourceVersion=0" (get nodes)" component="kube_client_runtime" source="kubernetes.go:75" 
2017-09-05T05:55:29.808330533Z time="2017-09-05T05:55:29Z" level=error msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:180: Failed to list *v1.Service: Forbidden: "/api/v1/services?resourceVersion=0" (get services)" component="kube_client_runtime" source="kubernetes.go:75" 
2017-09-05T05:55:30.345444610Z time="2017-09-05T05:55:30Z" level=error msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:227: Failed to list *v1.Service: Forbidden: "/api/v1/services?resourceVersion=0" (get services)" component="kube_client_runtime" source="kubernetes.go:75" 
2017-09-05T05:55:30.558481803Z time="2017-09-05T05:55:30Z" level=error msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:179: Failed to list *v1.Endpoints: Forbidden: "/api/v1/endpoints?resourceVersion=0" (get endpoints)" component="kube_client_runtime" source="kubernetes.go:75" 
2017-09-05T05:55:30.661439331Z time="2017-09-05T05:55:30Z" level=error msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:181: Failed to list *v1.Pod: Forbidden: "/api/v1/pods?resourceVersion=0" (get pods)" component="kube_client_runtime" source="kubernetes.go:75" 
2017-09-05T05:55:30.719157518Z time="2017-09-05T05:55:30Z" level=error msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:207: Failed to list *v1.Pod: Forbidden: "/api/v1/pods?resourceVersion=0" (get pods)" component="kube_client_runtime" source="kubernetes.go:75" 
2017-09-05T05:55:30.810896406Z time="2017-09-05T05:55:30Z" level=error msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:245: Failed to list *v1.Node: Forbidden: "/api/v1/nodes?resourceVersion=0" (get nodes)" component="kube_client_runtime" source="kubernetes.go:75" 
2017-09-05T05:55:30.810918023Z time="2017-09-05T05:55:30Z" level=error msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:180: Failed to list *v1.Service: Forbidden: "/api/v1/services?resourceVersion=0" (get services)" component="kube_client_runtime" source="kubernetes.go:75" 
2017-09-05T05:55:31.348096843Z time="2017-09-05T05:55:31Z" level=error msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:227: Failed to list *v1.Service: Forbidden: "/api/v1/services?resourceVersion=0" (get services)" component="kube_client_runtime" source="kubernetes.go:75" 
2017-09-05T05:55:31.561770991Z time="2017-09-05T05:55:31Z" level=error msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:179: Failed to list *v1.Endpoints: Forbidden: "/api/v1/endpoints?resourceVersion=0" (get endpoints)" component="kube_client_runtime" source="kubernetes.go:75" 
2017-09-05T05:55:31.665413249Z time="2017-09-05T05:55:31Z" level=error msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:181: Failed to list *v1.Pod: Forbidden: "/api/v1/pods?resourceVersion=0" (get pods)" component="kube_client_runtime" source="kubernetes.go:75" 
2017-09-05T05:55:31.721623025Z time="2017-09-05T05:55:31Z" level=error msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:207: Failed to list *v1.Pod: Forbidden: "/api/v1/pods?resourceVersion=0" (get pods)" component="kube_client_runtime" source="kubernetes.go:75" 
2017-09-05T05:55:31.813526530Z time="2017-09-05T05:55:31Z" level=error msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:245: Failed to list *v1.Node: Forbidden: "/api/v1/nodes?resourceVersion=0" (get nodes)" component="kube_client_runtime" source="kubernetes.go:75" 
2017-09-05T05:55:31.813547360Z time="2017-09-05T05:55:31Z" level=error msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:180: Failed to list *v1.Service: Forbidden: "/api/v1/services?resourceVersion=0" (get services)" component="kube_client_runtime" source="kubernetes.go:75" 
2017-09-05T05:55:32.350782045Z time="2017-09-05T05:55:32Z" level=error msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:227: Failed to list *v1.Service: Forbidden: "/api/v1/services?resourceVersion=0" (get services)" component="kube_client_runtime" source="kubernetes.go:75" 
2017-09-05T05:55:32.565790623Z time="2017-09-05T05:55:32Z" level=error msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:179: Failed to list *v1.Endpoints: Forbidden: "/api/v1/endpoints?resourceVersion=0" (get endpoints)" component="kube_client_runtime" source="kubernetes.go:75" 
2017-09-05T05:55:32.668112522Z time="2017-09-05T05:55:32Z" level=error msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:181: Failed to list *v1.Pod: Forbidden: "/api/v1/pods?resourceVersion=0" (get pods)" component="kube_client_runtime" source="kubernetes.go:75" 
2017-09-05T05:55:32.724119946Z time="2017-09-05T05:55:32Z" level=error msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:207: Failed to list *v1.Pod: Forbidden: "/api/v1/pods?resourceVersion=0" (get pods)" component="kube_client_runtime" source="kubernetes.go:75" 
2017-09-05T05:55:32.816956438Z time="2017-09-05T05:55:32Z" level=error msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:180: Failed to list *v1.Service: Forbidden: "/api/v1/services?resourceVersion=0" (get services)" component="kube_client_runtime" source="kubernetes.go:75"
zatricky commented

This is a permissions issue as a result of having an rbac-enabled cluster. You will see there was a recent commit to re-enable rbac configs #76. I haven't looked at it yet due to time constraints. See also my pull request. It does not enable rbac, merely supplements the commented rbac configs #73

trahloff commented

After pulling the mentioned PR it seems like the error is fixed - thanks!

Sadly there is a new problem: when applying the manifest-all.yml the error error validating "manifests-all.yaml": error validating data: the server could not find the requested resource; is thrown. Do you have any idea why?

rootsongjc commented

@trahloff What's the kubernetes version? I use kubernetes 1.6.0 with RBAC enabled get these error for kube-state-metrcs pod:

2017-09-25T11:53:22.578267576Z E0925 11:53:22.578043       1 reflector.go:201] k8s.io/kube-state-metrics/collectors/cronjob.go:86: Failed to list *v2alpha1.CronJob: User "system:serviceaccount:monitoring:kube-state-metrics" cannot list cronjobs.batch at the cluster scope. (get cronjobs.batch)
2017-09-25T11:53:22.579199644Z E0925 11:53:22.579002       1 reflector.go:201] k8s.io/kube-state-metrics/collectors/persistentvolumeclaim.go:60: Failed to list *v1.PersistentVolumeClaim: User "system:serviceaccount:monitoring:kube-state-metrics" cannot list persistentvolumeclaims at the cluster scope. (get persistentvolumeclaims)
2017-09-25T11:53:22.580366842Z E0925 11:53:22.580177       1 reflector.go:201] k8s.io/kube-state-metrics/collectors/statefulset.go:68: Failed to list *v1beta1.StatefulSet: User "system:serviceaccount:monitoring:kube-state-metrics" cannot list statefulsets.apps at the cluster scope. (get statefulsets.apps)
2017-09-25T11:53:23.578999887Z E0925 11:53:23.578734       1 reflector.go:201] k8s.io/kube-state-metrics/collectors/job.go:106: Failed to list *v1.Job: User "system:serviceaccount:monitoring:kube-state-metrics" cannot list jobs.batch at the cluster scope. (get jobs.batch)

And the job pod grafana-import-dashboards doesn't work.

gaya3chandran1 commented

@zatricky Despsite disabling RBAC I am still not getting pod specific metrics.

pipo02mix commented

It should be fixed now. I am closing, please feel free to open again if still happening