NFD logs crash approximately 7 hours after installation without touching the cluster
Closed this issue · 4 comments
Approximately 7 hours after NFD has been installed, the following error shows up in the NFD controller manager's logs:
E0903 02:52:07.981627 1 reflector.go:138] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:225: Failed to watch *v1.NodeFeatureDiscovery: the server has received too many requests and has asked us to try again later (get nodefeaturediscoveries.nfd.openshift.io)
E0903 02:52:10.566841 1 reflector.go:138] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:225: Failed to watch *v1.NodeFeatureDiscovery: the server has received too many requests and has asked us to try again later (get nodefeaturediscoveries.nfd.openshift.io)
This issue is easily repeatable by letting NFD run for several hours, such as overnight. I have reproduced it with and without using the cluster during that 7 hour period.
Seems like a controller manager issue?
Issues go stale after 90d of inactivity.
Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.
If this issue is safe to close now please do so with /close.
/lifecycle stale
Stale issues rot after 30d of inactivity.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
Exclude this issue from closing by commenting /lifecycle frozen.
If this issue is safe to close now please do so with /close.
/lifecycle rotten
/remove-lifecycle stale
Rotten issues close after 30d of inactivity.
Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.
/close
@openshift-bot: Closing this issue.
In response to this:
Rotten issues close after 30d of inactivity.
Reopen the issue by commenting
/reopen.
Mark the issue as fresh by commenting/remove-lifecycle rotten.
Exclude this issue from closing again by commenting/lifecycle frozen./close
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.