openshift-monitoring prometheus instance cannot access resources in openshift-logging namespace
4n4nd opened this issue · 5 comments
Describe the bug
openshift-monitoring prometheus instance cannot access resources in openshift-logging namespace
Environment
- ocp version: 4.8.x
- ClusterLogging instance: link
Logs
Logs from the openshift-monitoring prometheus instance:
level=error ts=2021-11-23T16:12:00.401Z caller=klog.go:96 component=k8s_client_runtime func=ErrorDepth msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:431: Failed to watch *v1.Pod: failed to list *v1.Pod: pods is forbidden: User \"system:serviceaccount:openshift-monitoring:prometheus-k8s\" cannot list resource \"pods\" in API group \"\" in the namespace \"openshift-logging\""
level=error ts=2021-11-23T16:12:14.897Z caller=klog.go:96 component=k8s_client_runtime func=ErrorDepth msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:430: Failed to watch *v1.Service: failed to list *v1.Service: services is forbidden: User \"system:serviceaccount:openshift-monitoring:prometheus-k8s\" cannot list resource \"services\" in API group \"\" in the namespace \"openshift-logging\""
level=error ts=2021-11-23T16:12:24.108Z caller=klog.go:96 component=k8s_client_runtime func=ErrorDepth msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:429: Failed to watch *v1.Endpoints: failed to list *v1.Endpoints: endpoints is forbidden: User \"system:serviceaccount:openshift-monitoring:prometheus-k8s\" cannot list resource \"endpoints\" in API group \"\" in the namespace \"openshift-logging\""
level=error ts=2021-11-23T16:12:55.558Z caller=klog.go:96 component=k8s_client_runtime func=ErrorDepth msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:431: Failed to watch *v1.Pod: failed to list *v1.Pod: pods is forbidden: User \"system:serviceaccount:openshift-monitoring:prometheus-k8s\" cannot list resource \"pods\" in API group \"\" in the namespace \"openshift-logging\""
level=error ts=2021-11-23T16:13:00.985Z caller=klog.go:96 component=k8s_client_runtime func=ErrorDepth msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:429: Failed to watch *v1.Endpoints: failed to list *v1.Endpoints: endpoints is forbidden: User \"system:serviceaccount:openshift-monitoring:prometheus-k8s\" cannot list resource \"endpoints\" in API group \"\" in the namespace \"openshift-logging\""
level=error ts=2021-11-23T16:13:02.670Z caller=klog.go:96 component=k8s_client_runtime func=ErrorDepth msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:430: Failed to watch *v1.Service: failed to list *v1.Service: services is forbidden: User \"system:serviceaccount:openshift-monitoring:prometheus-k8s\" cannot list resource \"services\" in API group \"\" in the namespace \"openshift-logging\""
Expected behavior
The required roles/bindings are created when the operator is installed and the recommended monitoring is enabled using these instructions.
Actual behavior
Roles/bindings are not created resulting in the errors with openshift-monitoring prometheus.
To Reproduce
Steps to reproduce the behavior:
- Follow steps here to install the CLO operator
- Go to
openshift-monitoringnamespace and check logs for any prometheus instances
Additional context
Related issue: operate-first/apps#1355
Current solution: Manually create the role/bindings (operate-first/apps#1407)
@vparfonov just rechecked, I am still seeing this issue. In PR #1272, I don't see any roles/bindings that would give the required permissions to the prometheus-k8s serviceaccount.
I just found the required role/binding in https://github.com/openshift/cluster-logging-operator/tree/master/manifests/5.4
Confirm I see the same issue on a fresh 4.8 cluster install with cluster-logging: 5.3.4-13 using the stable logging subscription. Seems like it's missing the clusterlogging-collector-metrics role and role binding.