Failed to watch secrets: Stream Error
shay-ul opened this issue · 0 comments
shay-ul commented
Describe the bug
We have a cluster with more than 10,000 secrets. Kubernetes replicator fails to replicate new secrets, and logs the following:
I0110 13:29:37.790193 1 trace.go:219] Trace[1458323237]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/client-go@v0.26.3/tools/cache/reflector.go:169 (10-Jan-2024 13:28:37.685) (total time: 60104ms):
Trace[1458323237]: ---"Objects listed" error:stream error when reading response body, may be caused by closed connection. Please retry. Original error: stream error: stream ID 31; INTERNAL_ERROR; received from peer 60104ms (13:29:37.790)
Trace[1458323237]: [1m0.104611175s] [1m0.104611175s] END
E0110 13:29:37.790213 1 reflector.go:140] pkg/mod/k8s.io/client-go@v0.26.3/tools/cache/reflector.go:169: Failed to watch *v1.Secret: failed to list *v1.Secret: stream error when reading response body, may be caused by closed connection. Please retry. Original error: stream error: stream ID 31; INTERNAL_ERROR; received from peer
Environment:
- Kubernetes version: EKS 1.24
- kubernetes-replicator version: v2.8.0
Additional context
We believe this is a timeout related issue. The API Server takes too long to respond (listing secrets across all namespace). If this is the case, we would like to be able to set a timeout for the secret list commands.
We also set higher CPU and Memory requests and limits. We are still facing this issue even when setting excessive requests and limits.