GoogleCloudPlatform/gke-managed-certs

FORBIDDEN error after certificate creation

lwsanty opened this issue · 6 comments

I'm creating ingresses with managed certificates as in example https://cloud.google.com/kubernetes-engine/docs/how-to/managed-certs

I even still have one running on subdomain1.domain.com
I also have been successfully creating ingresses for the other sub-domains, but today I faced this problem.

kubectl describe managedcertificate -n web-app
Name:         web-app-certificate
Namespace:    web-app
Labels:       <none>
Annotations:  <none>
API Version:  networking.gke.io/v1beta1
Kind:         ManagedCertificate
Metadata:
  Creation Timestamp:  2020-01-13T19:39:37Z
  Generation:          2
  Resource Version:    2270
  Self Link:           /apis/networking.gke.io/v1beta1/namespaces/web-app/managedcertificates/web-app-certificate
  UID:                 6ea7a4bd-363c-11ea-840c-42010af00146
Spec:
  Domains:
// here's 
    subdomain2.domain.com
Status:
  Certificate Name:  mcrt-cfb380b2-0b2c-4deb-b264-1e5be4ad259a
  Domain Status:
Events:
  Type     Reason        Age                   From                            Message
  ----     ------        ----                  ----                            -------
  Warning  BackendError  6m9s                  managed-certificate-controller  operation operation-1578944378860-59c0aa2d274a8-547f28d8-6dacdff0 failed: FORBIDDEN
  Warning  BackendError  5m58s                 managed-certificate-controller  operation operation-1578944390237-59c0aa3800bc9-25ad682d-099f8de1 failed: FORBIDDEN
  Warning  BackendError  5m47s                 managed-certificate-controller  operation operation-1578944401176-59c0aa426f7b3-13685221-86c3432c failed: FORBIDDEN
  Warning  BackendError  5m44s                 managed-certificate-controller  operation operation-1578944404387-59c0aa457f52d-456340a9-ecc77fa4 failed: FORBIDDEN
  Warning  BackendError  5m36s                 managed-certificate-controller  operation operation-1578944412291-59c0aa4d092b9-f667224d-1470767b failed: FORBIDDEN
  Warning  BackendError  5m24s                 managed-certificate-controller  operation operation-1578944424029-59c0aa583ad65-b073f0c1-a547e6a6 failed: FORBIDDEN
  Warning  BackendError  5m13s                 managed-certificate-controller  operation operation-1578944435216-59c0aa62e6263-3a24c18d-fe24c347 failed: FORBIDDEN
  Warning  BackendError  5m1s                  managed-certificate-controller  operation operation-1578944446746-59c0aa6de4dbb-bb645422-cdeb522c failed: FORBIDDEN
  Warning  BackendError  4m49s                 managed-certificate-controller  operation operation-1578944458846-59c0aa796f19f-4fd9164f-c53f59d8 failed: FORBIDDEN
  Warning  BackendError  16s (x18 over 4m36s)  managed-certificate-controller  (combined from similar events): operation operation-1578944731319-59c0ab7d48df7-733c60e6-604c77bd failed: FORBIDDEN

Is there any chance to know more details beyond the FORBIDDEN?

I have the exact same issue on a managed GKE cluster v1.14.8-gke.33

drcca commented

I've run into this before and the problem was that I ran into a QUOTA limit. Check SSL Cert quotas?

@drcca thank you for a reply!
Indeed back in the days when I submitted this issue the reason was in quotas eventually.

However, I have a doubt about closing this particular issue, because essentially it could be any other error and this commands' output does not provide any valuable insights.

Thanks for reporting this issue. The out-of-quota condition should be handled properly. I haven't yet had time to investigate it, but I have it on my list. Sorry I can't promise any time to take a look at it.

The bug is fixed in the newest release, v1.0.0 (not yet released in GKE).

@krzykwas is there a way to bypass the issue in the meantime?