kubernetes-retired/hierarchical-namespaces

Timeout errors due to some issue with the validating webhook

Closed this issue · 5 comments

Mais316 commented

Hi All,

We have hnc version 1.0.0 and our customers reported that they can no longer create any ns, checking the controller logs we can see

{"level":"error","ts":1688395630.3355894,"logger":"hierarchyconfig.reconcile","msg":"while updating apiserver","rid":3,"ns":"hnc-xyz","error":"Internal error occurred: failed calling webhook \"hierarchyconfigurations.hnc.x-k8s.io\": failed to call webhook: Post \"https://hnc-webhook-service.hnc-system.svc:443/validate-hnc-x-k8s-io-v1alpha2-hierarchyconfigurations?timeout=30s\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"}
{"level":"error","ts":1688395630.335623,"logger":"controller.hierarchyconfiguration","msg":"Reconciler error","reconciler group":"hnc.x-k8s.io","reconciler kind":"HierarchyConfiguration","name":"hierarchy","namespace":"hnc-xyz","error":"Internal error occurred: failed calling webhook \"hierarchyconfigurations.hnc.x-k8s.io\": failed to call webhook: Post \"https://hnc-webhook-service.hnc-system.svc:443/validate-hnc-x-k8s-io-v1alpha2-hierarchyconfigurations?timeout=30s\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"}
{"level":"error","ts":1688395630.3359044,"logger":"hierarchyconfig.reconcile","msg":"while updating apiserver","rid":1,"ns":"hnc-abc","error":"Internal error occurred: failed calling webhook \"objects.hnc.x-k8s.io\": failed to call webhook: Post \"https://hnc-webhook-service.hnc-system.svc:443/validate-objects?timeout=30s\": context deadline exceeded"}
{"level":"error","ts":1688395630.3359256,"logger":"controller.hierarchyconfiguration","msg":"Reconciler error","reconciler group":"hnc.x-k8s.io","reconciler kind":"HierarchyConfiguration","name":"hierarchy","namespace":"hnc-abc","error":"Internal error occurred: failed calling webhook \"objects.hnc.x-k8s.io\": failed to call webhook: Post \"https://hnc-webhook-service.hnc-system.svc:443/validate-objects?timeout=30s\": context deadline exceeded"}

We tried below, nothing helped

  • Increasing the resources
  • timeouts

We check calico logs all looks find, then tried port-forward and we can curl to the service.
Kubernetes version is 1.124.12
Cluster is aws-eks, managed by Rancher 2.7.3

Appreciate your help as we are blocked, workaround is to remove the validation webhook, but that is not a solution.

k8s-triage-robot commented

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue as fresh with /remove-lifecycle stale
  • Close this issue with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

terricain commented

@Mais316 I encountered this just yesterday. My issue was (with AWS EKS) that the API sever in the control plane couldn't reach the webhook on 9443.

Does you're API server logs show anything? As I also saw lots of failed to call webhook errors in the API server logs. (Should have checked that first as went down a rabbit hole of running tcpdump to check if the manager was contacting the webhook and timing out first 😄)

k8s-triage-robot commented

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue as fresh with /remove-lifecycle rotten
  • Close this issue with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

k8s-triage-robot commented

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Reopen this issue with /reopen
  • Mark this issue as fresh with /remove-lifecycle rotten
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

k8s-ci-robot commented

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Reopen this issue with /reopen
  • Mark this issue as fresh with /remove-lifecycle rotten
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.