New custom error pages issue: Application Gateway Custom Error Status Code Is Invalid

[yevhenii0bozhenko]

Describe the bug
Ingress controller is unable to update Application Gateway (e.g. new pods IP addresses, backend pools..) if there are (in preview) custom error pages configured (codes: 403, 502).
The App Gateway feature is described here

I have custom error pages set on a Global Level (not on listeners one).

Ingress controller pods produce errors listed in kubectl logs section below.

Questions

• Is there any plans for these custom error codes to be supported by Kubernetes Ingress or any way to make it work with this case?
• What are the dependencies? Go modules or Azure APIs etc.

To Reproduce
Steps to reproduce the behavior:

1. Set the custom error page on a Global Level according to this official instruction here
2. Restart your deployment that has ingress pointing to it
3. Check the ingress controller pod logs, Application Gateway Activity Log, Backend Pool configuration of the related deployment for errors.

Ingress Controller details

• Output of kubectl describe pod <ingress controller> . The pod name can be obtained by running helm list.

Name:             ingress-appgw-deployment-857b4bbd5d-wxkqr
Namespace:        kube-system
Priority:         0
Service Account:  ingress-appgw-sa
Node:             aks-default-XXX/10.1.20.122
Start Time:       Wed, 16 Aug 2023 11:06:57 +0300
Labels:           app=ingress-appgw
                  kubernetes.azure.com/managedby=aks
                  pod-template-hash=857b4bbd5d
Annotations:      checksum/config: e0809d2c92ee39a97ffaeeb8829fd946c25f6fbb7fbba0e1d5d5911c69b0125d
                  cluster-autoscaler.kubernetes.io/safe-to-evict: true
                  kubernetes.azure.com/metrics-scrape: true
                  prometheus.io/path: /metrics
                  prometheus.io/port: 8123
                  prometheus.io/scrape: true
                  resource-id:
                    /subscriptions/XXX/resourceGroups/XXX/providers/Microsoft.ContainerService/managedClusters/b...
Status:           Running
IP:               10.1.20.137
IPs:
  IP:           10.1.20.137
Controlled By:  ReplicaSet/ingress-appgw-deployment-857b4bbd5d
Containers:
  ingress-appgw-container:
    Container ID:   containerd://1fcea36af6ffef7424710f786570c772357a779041dc0c4e488920c387c9ec65
    Image:          mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.5.3
    Image ID:       mcr.microsoft.com/azure-application-gateway/kubernetes-ingress@sha256:efd474df2c22af6e5abc201eb417098acec988feb70a763bb0782eda743fff42
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Wed, 16 Aug 2023 11:07:07 +0300
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     700m
      memory:  600Mi
    Requests:
      cpu:      100m
      memory:   20Mi
    Liveness:   http-get http://:8123/health/alive delay=15s timeout=1s period=20s #success=1 #failure=3
    Readiness:  http-get http://:8123/health/ready delay=5s timeout=1s period=10s #success=1 #failure=3
    Environment Variables from:
      ingress-appgw-cm  ConfigMap  Optional: false
    Environment:
      AGIC_POD_NAMESPACE:             kube-system (v1:metadata.namespace)
      KUBERNETES_PORT_443_TCP_ADDR:   XXX.azmk8s.io
      KUBERNETES_PORT:                tcp://XXX.azmk8s.io:443
      KUBERNETES_PORT_443_TCP:        tcp://XXX.azmk8s.io:443
      KUBERNETES_SERVICE_HOST:        XXX.azmk8s.io
      AZURE_CLOUD_PROVIDER_LOCATION:  /etc/kubernetes/azure.json
      AGIC_POD_NAME:                  ingress-appgw-deployment-857b4bbd5d-wxkqr (v1:metadata.name)
    Mounts:
      /etc/kubernetes/azure.json from cloud-provider-config (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-t2kqr (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  cloud-provider-config:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/kubernetes/azure.json
    HostPathType:  File
  kube-api-access-t2kqr:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 CriticalAddonsOnly op=Exists
                             node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:                      <none>

• Output of `kubectl logs .

I1106 08:30:55.003540       1 mutate_app_gateway.go:177] BEGIN AppGateway deployment
I1106 08:30:56.454097       1 mutate_app_gateway.go:183] END AppGateway deployment
E1106 08:30:56.454179       1 controller.go:141] network.ApplicationGatewaysClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code="ApplicationGatewayCustomErrorStatusCodeIsInvalid" Message="Custom error page has an invalid HTTP status code '500', for applicationGateways /subscriptions/XXX/resourceGroups/XXX/providers/Microsoft.Network/applicationGateways/XXX. Valid status codes are: '[403, 502]'." Details=[]
E1106 08:30:56.454192       1 worker.go:62] Error processing event.network.ApplicationGatewaysClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code="ApplicationGatewayCustomErrorStatusCodeIsInvalid" Message="Custom error page has an invalid HTTP status code '500', for applicationGateways /subscriptions/XXX/resourceGroups/XXX/providers/Microsoft.Network/applicationGateways/XXX. Valid status codes are: '[403, 502]'." Details=[]
I1106 08:30:56.454315       1 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"kube-system", Name:"ingress-appgw-deployment-857b4bbd5d-wxkqr", UID:"XXX", APIVersion:"v1", ResourceVersion:"140294170", FieldPath:""}): type: 'Warning' reason: 'FailedApplyingAppGwConfig' network.ApplicationGatewaysClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code="ApplicationGatewayCustomErrorStatusCodeIsInvalid" Message="Custom error page has an invalid HTTP status code '500', for applicationGateways /subscriptions/XXX/resourceGroups/XXX/providers/Microsoft.Network/applicationGateways/XXX. Valid status codes are: '[403, 502]'." Details=[]

• Any Azure support tickets associated with this issue.

This issue is related to issue #733 and PR #1350 but exposes another use case.

[akshaysngupta]

@yevhenii0bozhenko I suspect that this is happening because AGIC is using older azure sdk.