failed to wait for compliance caches
yuriydzobak opened this issue · 11 comments
yuriydzobak commented
Hi,
I don't have enabled the compliance in trivy operator
[INFO] ConfigAuditReports enabled
[INFO] VulnerabilityReports enabled
[INFO] RbacAssessmentReports enabled
[INFO] ExposedSecretReports enabled
[INFO] CISKubeBenchReports enabled
[INFO] InfraAssessmentReportClient enabled
but the application is crashed
Error: failed to wait for compliance caches to sync: timed out waiting for cache to be synced
Usage:
trivy-operator-polr-adapter run [flags]
Flags:
-c, --config string target configuration file
--enable-compliance Enable the transformation of ClusterComplianceDetailReports into ClusterPolicyReports
--enable-config-audit Enable the transformation of ConfigAuditReports into PolicyReports
--enable-exposed-secrets Enable the transformation of ExposedSecretReports into PolicyReports
--enable-infra-assessment Enable the transformation of InfraAssessmentReports into PolicyReports
--enable-kube-bench Enable the transformation of CISKubeBenchReports into ClusterPolicyReports
--enable-rbac-assessment Enable the transformation of RbacAssessmentReports into PolicyReports
--enable-vulnerability Enable the transformation of VulnerabilityReports into PolicyReports
-h, --help help for run
-k, --kubeconfig string absolute path to the kubeconfig file
failed to wait for compliance caches to sync: timed out waiting for cache to be synced
fjogeleit commented
Hey, did you check if the clustercompliancereport CRD is installed in your cluster? Depending on your version of the trivy operator it is maybe not available.
yuriydzobak commented
clustercompliancereport
it didn't install but i turned off in values.yaml but the app is still failing
adapters:
vulnerabilityReports:
enabled: true
# apply labels from the source report
applyLabels: []
configAuditReports:
enabled: true
applyLabels: []
cisKubeBenchReports:
enabled: true
applyLabels: []
complianceReports:
enabled: false
applyLabels: []
rbacAssessmentReports:
enabled: true
applyLabels: []
exposedSecretReports:
enabled: true
applyLabels: []
infraAssessmentReports:
enabled: true
applyLabels: []fjogeleit commented
Ah okay, then I misunderstood your question. I will have a look.
So it still tries to read compliancereports with complianceReports.enabled set to false?
yuriydzobak commented
Ah okay, then I misunderstood your question. I will have a look.
You understood correct, I changed description, sorry
So it still tries to read compliancereports with
complianceReports.enabledset to false?
yes, it's still
trivy-operator-adapter-5b849f8bc6-kml6s 1/1 Running 3 (80s ago) 7m46s[INFO] ConfigAuditReports enabled
[INFO] VulnerabilityReports enabled
[INFO] RbacAssessmentReports enabled
[INFO] ExposedSecretReports enabled
[INFO] CISKubeBenchReports enabled
[INFO] InfraAssessmentReportClient enabled
W1228 14:23:44.198888 1 reflector.go:424] pkg/mod/k8s.io/client-go@v0.26.0/tools/cache/reflector.go:169: failed to list *v1alpha1.InfraAssessmentReport: infraassessmentreports.aquasecurity.github.io is forbidden: User "system:serviceaccount:monitoring:trivy-operator-adapter" cannot list resource "infraassessmentreports" in API group "aquasecurity.github.io" at the cluster scope
E1228 14:23:44.198930 1 reflector.go:140] pkg/mod/k8s.io/client-go@v0.26.0/tools/cache/reflector.go:169: Failed to watch *v1alpha1.InfraAssessmentReport: failed to list *v1alpha1.InfraAssessmentReport: infraassessmentreports.aquasecurity.github.io is forbidden: User "system:serviceaccount:monitoring:trivy-operator-adapter" cannot list resource "infraassessmentreports" in API group "aquasecurity.github.io" at the cluster scope
W1228 14:23:45.660785 1 reflector.go:424] pkg/mod/k8s.io/client-go@v0.26.0/tools/cache/reflector.go:169: failed to list *v1alpha1.InfraAssessmentReport: infraassessmentreports.aquasecurity.github.io is forbidden: User "system:serviceaccount:monitoring:trivy-operator-adapter" cannot list resource "infraassessmentreports" in API group "aquasecurity.github.io" at the cluster scope
E1228 14:23:45.660814 1 reflector.go:140] pkg/mod/k8s.io/client-go@v0.26.0/tools/cache/reflector.go:169: Failed to watch *v1alpha1.InfraAssessmentReport: failed to list *v1alpha1.InfraAssessmentReport: infraassessmentreports.aquasecurity.github.io is forbidden: User "system:serviceaccount:monitoring:trivy-operator-adapter" cannot list resource "infraassessmentreports" in API group "aquasecurity.github.io" at the cluster scope
I1228 14:23:46.585157 1 request.go:682] Waited for 1.001956396s due to client-side throttling, not priority and fairness, request: POST:https://10.11.0.1:443/apis/wgpolicyk8s.io/v1alpha2/namespaces/kubeshark/policyreports
W1228 14:23:48.513899 1 reflector.go:424] pkg/mod/k8s.io/client-go@v0.26.0/tools/cache/reflector.go:169: failed to list *v1alpha1.InfraAssessmentReport: infraassessmentreports.aquasecurity.github.io is forbidden: User "system:serviceaccount:monitoring:trivy-operator-adapter" cannot list resource "infraassessmentreports" in API group "aquasecurity.github.io" at the cluster scope
E1228 14:23:48.514308 1 reflector.go:140] pkg/mod/k8s.io/client-go@v0.26.0/tools/cache/reflector.go:169: Failed to watch *v1alpha1.InfraAssessmentReport: failed to list *v1alpha1.InfraAssessmentReport: infraassessmentreports.aquasecurity.github.io is forbidden: User "system:serviceaccount:monitoring:trivy-operator-adapter" cannot list resource "infraassessmentreports" in API group "aquasecurity.github.io" at the cluster scope
W1228 14:23:54.984785 1 reflector.go:424] pkg/mod/k8s.io/client-go@v0.26.0/tools/cache/reflector.go:169: failed to list *v1alpha1.InfraAssessmentReport: infraassessmentreports.aquasecurity.github.io is forbidden: User "system:serviceaccount:monitoring:trivy-operator-adapter" cannot list resource "infraassessmentreports" in API group "aquasecurity.github.io" at the cluster scope
E1228 14:23:54.984824 1 reflector.go:140] pkg/mod/k8s.io/client-go@v0.26.0/tools/cache/reflector.go:169: Failed to watch *v1alpha1.InfraAssessmentReport: failed to list *v1alpha1.InfraAssessmentReport: infraassessmentreports.aquasecurity.github.io is forbidden: User "system:serviceaccount:monitoring:trivy-operator-adapter" cannot list resource "infraassessmentreports" in API group "aquasecurity.github.io" at the cluster scope
I1228 14:23:58.782402 1 request.go:682] Waited for 1.000297578s due to client-side throttling, not priority and fairness, request: GET:https://10.11.0.1:443/apis/wgpolicyk8s.io/v1alpha2/clusterpolicyreports/trivy-rbac-cpolr-clusterrole-679f75d6b5
W1228 14:24:06.809104 1 reflector.go:424] pkg/mod/k8s.io/client-go@v0.26.0/tools/cache/reflector.go:169: failed to list *v1alpha1.InfraAssessmentReport: infraassessmentreports.aquasecurity.github.io is forbidden: User "system:serviceaccount:monitoring:trivy-operator-adapter" cannot list resource "infraassessmentreports" in API group "aquasecurity.github.io" at the cluster scope
E1228 14:24:06.809133 1 reflector.go:140] pkg/mod/k8s.io/client-go@v0.26.0/tools/cache/reflector.go:169: Failed to watch *v1alpha1.InfraAssessmentReport: failed to list *v1alpha1.InfraAssessmentReport: infraassessmentreports.aquasecurity.github.io is forbidden: User "system:serviceaccount:monitoring:trivy-operator-adapter" cannot list resource "infraassessmentreports" in API group "aquasecurity.github.io" at the cluster scope
I1228 14:24:14.785460 1 request.go:682] Waited for 1.003264357s due to client-side throttling, not priority and fairness, request: GET:https://10.11.0.1:443/apis/wgpolicyk8s.io/v1alpha2/clusterpolicyreports/trivy-rbac-cpolr-clusterrole-strimzi-kafka-client
W1228 14:24:32.403984 1 reflector.go:424] pkg/mod/k8s.io/client-go@v0.26.0/tools/cache/reflector.go:169: failed to list *v1alpha1.InfraAssessmentReport: infraassessmentreports.aquasecurity.github.io is forbidden: User "system:serviceaccount:monitoring:trivy-operator-adapter" cannot list resource "infraassessmentreports" in API group "aquasecurity.github.io" at the cluster scope
E1228 14:24:32.404012 1 reflector.go:140] pkg/mod/k8s.io/client-go@v0.26.0/tools/cache/reflector.go:169: Failed to watch *v1alpha1.InfraAssessmentReport: failed to list *v1alpha1.InfraAssessmentReport: infraassessmentreports.aquasecurity.github.io is forbidden: User "system:serviceaccount:monitoring:trivy-operator-adapter" cannot list resource "infraassessmentreports" in API group "aquasecurity.github.io" at the cluster scope
I1228 14:24:35.851365 1 request.go:682] Waited for 1.163116056s due to client-side throttling, not priority and fairness, request: GET:https://10.11.0.1:443/apis/discovery.k8s.io/v1beta1?timeout=32s
I1228 14:24:45.851449 1 request.go:682] Waited for 1.156883927s due to client-side throttling, not priority and fairness, request: GET:https://10.11.0.1:443/apis/batch/v1beta1?timeout=32s
I1228 14:24:56.051268 1 request.go:682] Waited for 11.356236949s due to client-side throttling, not priority and fairness, request: GET:https://10.11.0.1:443/apis/snapshot.storage.k8s.io/v1?timeout=32s
I1228 14:25:06.051717 1 request.go:682] Waited for 1.364532139s due to client-side throttling, not priority and fairness, request: GET:https://10.11.0.1:443/apis/rbac.authorization.k8s.io/v1?timeout=32s
I1228 14:25:16.251319 1 request.go:682] Waited for 1.564216855s due to client-side throttling, not priority and fairness, request: GET:https://10.11.0.1:443/apis/certificates.k8s.io/v1?timeout=32s
W1228 14:25:18.680478 1 reflector.go:424] pkg/mod/k8s.io/client-go@v0.26.0/tools/cache/reflector.go:169: failed to list *v1alpha1.InfraAssessmentReport: infraassessmentreports.aquasecurity.github.io is forbidden: User "system:serviceaccount:monitoring:trivy-operator-adapter" cannot list resource "infraassessmentreports" in API group "aquasecurity.github.io" at the cluster scope
E1228 14:25:18.680512 1 reflector.go:140] pkg/mod/k8s.io/client-go@v0.26.0/tools/cache/reflector.go:169: Failed to watch *v1alpha1.InfraAssessmentReport: failed to list *v1alpha1.InfraAssessmentReport: infraassessmentreports.aquasecurity.github.io is forbidden: User "system:serviceaccount:monitoring:trivy-operator-adapter" cannot list resource "infraassessmentreports" in API group "aquasecurity.github.io" at the cluster scope
I1228 14:25:26.451744 1 request.go:682] Waited for 11.76386814s due to client-side throttling, not priority and fairness, request: GET:https://10.11.0.1:443/apis/isindir.github.com/v1alpha2?timeout=32s
I1228 14:25:36.651437 1 request.go:682] Waited for 1.964975766s due to client-side throttling, not priority and fairness, request: GET:https://10.11.0.1:443/apis/generators.external-secrets.io/v1alpha1?timeout=32s
Error: failed to wait for configaudit caches to sync: timed out waiting for cache to be synced
Usage:
trivy-operator-polr-adapter run [flags]
Flags:
-c, --config string target configuration file
--enable-compliance Enable the transformation of ClusterComplianceDetailReports into ClusterPolicyReports
--enable-config-audit Enable the transformation of ConfigAuditReports into PolicyReports
--enable-exposed-secrets Enable the transformation of ExposedSecretReports into PolicyReports
--enable-infra-assessment Enable the transformation of InfraAssessmentReports into PolicyReports
--enable-kube-bench Enable the transformation of CISKubeBenchReports into ClusterPolicyReports
--enable-rbac-assessment Enable the transformation of RbacAssessmentReports into PolicyReports
--enable-vulnerability Enable the transformation of VulnerabilityReports into PolicyReports
-h, --help help for run
-k, --kubeconfig string absolute path to the kubeconfig file
failed to wait for configaudit caches to sync: timed out waiting for cache to be syncedfjogeleit commented
In this errors are the failed caches are configaudit, which are enabled. How large is your cluster? The only possible solution I could found for now would be to add a configuration for the timeout and increase it if needed. Could be possible that the default of 2 minutes are not enough for a higher amount of Trivy CRD Resources.
yuriydzobak commented
I've disabled it too
helm get values trivy-operator-adapter dc06-ldap[12:57:39]
USER-SUPPLIED VALUES:
adapters:
cisKubeBenchReports:
enabled: true
complianceReports:
enabled: false
configAuditReports:
enabled: false
exposedSecretReports:
enabled: false
infraAssessmentReports:
enabled: false
rbacAssessmentReports:
enabled: false
vulnerabilityReports:
enabled: true
fullnameOverride: trivy-operator-adapter
nodeSelector:
group-name: worker-group-infra
podAnnotations: {}
resources:
limits:
cpu: 100m
memory: 128Mi
requests:
cpu: 100m
memory: 128MiControlled By: ReplicaSet/trivy-operator-adapter-5b849f8bc6
Containers:
trivy-operator-polr-adapter:
Container ID: containerd://2fea4717def8ab8cae27f48f3acf1906a573b2de783c6f1461856ad4c70792fe
Image: ghcr.io/fjogeleit/trivy-operator-polr-adapter:0.3.1
Image ID: ghcr.io/fjogeleit/trivy-operator-polr-adapter@sha256:f270baae7515b261c82be5de39aac38c14d75c70bcf5cc1afb820b0ed91f265f
Port: <none>
Host Port: <none>
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Thu, 29 Dec 2022 12:54:28 +0200
Finished: Thu, 29 Dec 2022 12:56:29 +0200
Ready: False
Restart Count: 16
Limits:
cpu: 100m
memory: 128Mi
Requests:
cpu: 100m
memory: 128Mi
Readiness: exec [/app/trivy-operator-polr-adapter version] delay=0s timeout=1s period=10s #success=1 #failure=3
Environment: <none>
Mounts:
/app/config.yaml from config-file (ro,path="config.yaml")
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-s8gzt (ro)trivy-operator-adapter-5b849f8bc6-5j75h 1/1 Running 16 (6m51s ago) 91m
[INFO] VulnerabilityReports enabled
[INFO] CISKubeBenchReports enabled
I1229 10:48:17.221998 1 request.go:682] Waited for 1.100157662s due to client-side throttling, not priority and fairness, request: GET:https://10.11.0.1:443/apis/kyverno.io/v1beta1?timeout=32s
I1229 10:48:27.421650 1 request.go:682] Waited for 1.300377222s due to client-side throttling, not priority and fairness, request: GET:https://10.11.0.1:443/apis/flowcontrol.apiserver.k8s.io/v1beta2?timeout=32s
I1229 10:48:37.422086 1 request.go:682] Waited for 11.300079533s due to client-side throttling, not priority and fairness, request: GET:https://10.11.0.1:443/apis/helm.toolkit.fluxcd.io/v2beta1?timeout=32s
I1229 10:48:47.622159 1 request.go:682] Waited for 1.501450994s due to client-side throttling, not priority and fairness, request: GET:https://10.11.0.1:443/apis/events.k8s.io/v1?timeout=32s
I1229 10:48:57.622347 1 request.go:682] Waited for 1.501142727s due to client-side throttling, not priority and fairness, request: GET:https://10.11.0.1:443/apis/admissionregistration.k8s.io/v1?timeout=32s
I1229 10:49:07.823592 1 request.go:682] Waited for 11.702036764s due to client-side throttling, not priority and fairness, request: GET:https://10.11.0.1:443/apis/core.strimzi.io/v1beta2?timeout=32s
I1229 10:49:18.022073 1 request.go:682] Waited for 1.901679701s due to client-side throttling, not priority and fairness, request: GET:https://10.11.0.1:443/apis/flowcontrol.apiserver.k8s.io/v1beta1?timeout=32s
Error: failed to wait for ciskubebench caches to sync: timed out waiting for cache to be synced
Usage:
trivy-operator-polr-adapter run [flags]
Flags:
-c, --config string target configuration file
--enable-compliance Enable the transformation of ClusterComplianceDetailReports into ClusterPolicyReports
--enable-config-audit Enable the transformation of ConfigAuditReports into PolicyReports
--enable-exposed-secrets Enable the transformation of ExposedSecretReports into PolicyReports
--enable-infra-assessment Enable the transformation of InfraAssessmentReports into PolicyReports
--enable-kube-bench Enable the transformation of CISKubeBenchReports into ClusterPolicyReports
--enable-rbac-assessment Enable the transformation of RbacAssessmentReports into PolicyReports
--enable-vulnerability Enable the transformation of VulnerabilityReports into PolicyReports
-h, --help help for run
-k, --kubeconfig string absolute path to the kubeconfig file
failed to wait for ciskubebench caches to sync: timed out waiting for cache to be synced
but the issue is still existing
The cluster is not so huge, just 140 PODs and 13 nodes
fjogeleit commented
Do you have the cis kube bench CRD installed? This CRD is not available in newer trivy operator versions.
fjogeleit commented
Does it also appear with only vulnerability reports enabled for example?
yuriydzobak commented
I don't have cis kube
When i disabled it seems the application is working fine
trivy-operator-adapter-5b849f8bc6-vmzxz 1/1 Running 0 2m38s
$ klf trivy-operator-adapter-5b849f8bc6-vmzxz dc06-ldap[14:54:46]
[INFO] VulnerabilityReports enabledSeems the issue was in this option
Thank you!
yuriydzobak commented
I think, maybe disable the options in values.yaml and enable if it's necessary for user. That helps to reduce miss configuration
Also, I wish you Happy New Year!
Thanks
fjogeleit commented
Okay great. CIS Kube Bench is disabled by default and the readme has a deprecation info but I will improve it in the cli output as well.