Kong/kubernetes-ingress-controller

HTTPRoute status is reconciled regardless of `Gateway`'s `AllowedRoutes` and/or `--gateway-to-reconcile` flag

Closed this issue · 0 comments

pmalek commented

Is there an existing issue for this?

  • I have searched the existing issues

Current Behavior

When more than 1 KIC is deployed (e.g. using KGO and Gateway resources) then HTTPRoutes that are to be reconciled have their status filled in regardless of the listeners' AllowedRoutes and/or --gateway-to-reconcile flag.

This issue has 2 aspects:

  • Not filtering routes per their Gateway attachment (consulted with --gateway-to-reconcile flag) does make sense to to properly react to situations where an attachment of a route changes (a route was attached to a gateway marked with the flag but now it's not), this could be narrowed down to filtering update events when the route:

    • was created attached to a Gateway that is being reconciled
    • becomes attached to a Gateway that is being reconciled
    • becomes detached from a Gateway that is being reconciled: this should cause the removal of the route from dataplane's configuration

  • Because of the above lack of filtering and KIC clearing the route status only matching on controller name (and not taking into account listeners AllowedRoutes) this results in perpetual loop of

    2024-05-22T12:26:12Z    error   controllers.HTTPRoute   Reconciler error        {"reconcileID": "9418485c-120c-4c51-a2ba-e8a13f648289", "error": "Operation cannot be fulfilled on httproutes.gateway.networking.k8s.io \"httproute-echo\": the object has been modified; please apply your changes to the latest version and try again"}
2024-05-22T12:26:12Z    debug   controllers.HTTPRoute   Processing httproute    {"GatewayV1HTTPRoute": {"name":"httproute-echo","namespace":"default"}, "v": 1, "namespace": "default", "name": "httproute-echo"}
2024-05-22T12:26:12Z    debug   controllers.HTTPRoute   Checking deletion timestamp     {"GatewayV1HTTPRoute": {"name":"httproute-echo","namespace":"default"}, "v": 1, "namespace": "default", "name": "httproute-echo"}
2024-05-22T12:26:12Z    debug   controllers.HTTPRoute   Retrieving GatewayClass and Gateway for route   {"GatewayV1HTTPRoute": {"name":"httproute-echo","namespace":"default"}, "v": 1, "namespace": "default", "name": "httproute-echo"}
2024-05-22T12:26:12Z    debug   controllers.HTTPRoute   Unsupported route found, processing to verify whether it was ever supported     {"GatewayV1HTTPRoute": {"name":"httproute-echo","namespace":"default"}, "v": 1, "namespace": "default", "name": "httproute-echo"}
2024-05-22T12:26:12Z    debug   controllers.HTTPRoute   Unsupported route was previously supported, status was updated  {"GatewayV1HTTPRoute": {"name":"httproute-echo","namespace":"default"}, "v": 1, "namespace": "default", "name": "httproute-echo"}
2024-05-22T12:26:12Z    debug   controllers.Gateway     The request does not match the specified Gateway and will be skipped.   {"v": 1, "gateway": "default/kong2"}
2024-05-22T12:26:12Z    debug   controllers.HTTPRoute   Processing httproute    {"GatewayV1HTTPRoute": {"name":"httproute-echo","namespace":"default"}, "v": 1, "namespace": "default", "name": "httproute-echo"}
2024-05-22T12:26:12Z    debug   controllers.HTTPRoute   Checking deletion timestamp     {"GatewayV1HTTPRoute": {"name":"httproute-echo","namespace":"default"}, "v": 1, "namespace": "default", "name": "httproute-echo"}
2024-05-22T12:26:12Z    debug   controllers.HTTPRoute   Retrieving GatewayClass and Gateway for route   {"GatewayV1HTTPRoute": {"name":"httproute-echo","namespace":"default"}, "v": 1, "namespace": "default", "name": "httproute-echo"}
2024-05-22T12:26:12Z    debug   controllers.HTTPRoute   Unsupported route found, processing to verify whether it was ever supported     {"GatewayV1HTTPRoute": {"name":"httproute-echo","namespace":"default"}, "v": 1, "namespace": "default", "name": "httproute-echo"}
2024-05-22T12:26:12Z    debug   controllers.HTTPRoute   Ensuring that dataplane is updated to remove unsupported route (if applicable)  {"GatewayV1HTTPRoute": {"name":"httproute-echo","namespace":"default"}, "v": 1, "namespace": "default", "name": "httproute-echo"}
2024-05-22T12:26:12Z    debug   controllers.HTTPRoute   Processing httproute    {"GatewayV1HTTPRoute": {"name":"httproute-echo","namespace":"default"}, "v": 1, "namespace": "default", "name": "httproute-echo"}
2024-05-22T12:26:12Z    debug   controllers.HTTPRoute   Checking deletion timestamp     {"GatewayV1HTTPRoute": {"name":"httproute-echo","namespace":"default"}, "v": 1, "namespace": "default", "name": "httproute-echo"}
2024-05-22T12:26:12Z    debug   controllers.HTTPRoute   Retrieving GatewayClass and Gateway for route   {"GatewayV1HTTPRoute": {"name":"httproute-echo","namespace":"default"}, "v": 1, "namespace": "default", "name": "httproute-echo"}
2024-05-22T12:26:12Z    debug   controllers.HTTPRoute   Unsupported route found, processing to verify whether it was ever supported     {"GatewayV1HTTPRoute": {"name":"httproute-echo","namespace":"default"}, "v": 1, "namespace": "default", "name": "httproute-echo"}
2024-05-22T12:26:12Z    debug   controllers.HTTPRoute   Ensuring that dataplane is updated to remove unsupported route (if applicable)  {"GatewayV1HTTPRoute": {"name":"httproute-echo","namespace":"default"}, "v": 1, "namespace": "default", "name": "httproute-echo"}

This also comes up when running KGO Gateway API conformance tests because that deploys multiple Gateways (and thus KIC instances).

Expected Behavior

When more than 1 KIC is deployed with --gateway-to-reconcile, and that Gateway has allowedRoutes set, then that is taken into account when setting/clearing routes' statuses.

Steps To Reproduce

Using KGO: 

1. Deploy KGO 
1. Deploy the following manifests:


apiVersion: v1
kind: Service
metadata:
  name: echo
spec:
  ports:
    - protocol: TCP
      name: http
      port: 80
      targetPort: http
  selector:
    app: echo
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: echo
  name: echo
spec:
  replicas: 1
  selector:
    matchLabels:
      app: echo
  template:
    metadata:
      labels:
        app: echo
    spec:
      containers:
        - name: echo
          image: registry.k8s.io/e2e-test-images/agnhost:2.40
          command:
            - /agnhost
            - netexec
            - --http-port=8080
          ports:
            - containerPort: 8080
              name: http
          env:
            - name: NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            - name: POD_IP
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP
          resources:
            requests:
              cpu: 10m
---
kind: GatewayConfiguration
apiVersion: gateway-operator.konghq.com/v1beta1
metadata:
  name: kong
  namespace: default
spec:
  dataPlaneOptions:
    deployment:
      podTemplateSpec:
        spec:
          containers:
          - name: proxy
            # renovate: datasource=docker versioning=docker
            image: kong/kong-gateway:3.6
            readinessProbe:
              initialDelaySeconds: 1
              periodSeconds: 1
  controlPlaneOptions:
    deployment:
      podTemplateSpec:
        spec:
          containers:
          - name: controller
            # renovate: datasource=docker versioning=docker
            image: kong/kubernetes-ingress-controller:3.1.5
            readinessProbe:
              initialDelaySeconds: 1
              periodSeconds: 1
            env:
            - name: CONTROLLER_LOG_LEVEL
              value: debug
---
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: kong
spec:
  controllerName: konghq.com/gateway-operator
  parametersRef:
    group: gateway-operator.konghq.com
    kind: GatewayConfiguration
    name: kong
    namespace: default
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: kong
  namespace: default
spec:
  gatewayClassName: kong
  listeners:
  - name: http
    protocol: HTTP
    port: 80
    allowedRoutes:
      kinds:
      - kind: HTTPRoute
      namespaces:
        from: Selector
        selector:
          matchLabels:
            # This label is added automatically as of K8s 1.22 to all namespaces
            kubernetes.io/metadata.name: default
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: httproute-echo
  namespace: default
  annotations:
    konghq.com/strip-path: "true"
spec:
  parentRefs:
  - name: kong
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /echo
    backendRefs:
    - name: echo
      kind: Service
      port: 80
---
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: kong2
spec:
  controllerName: konghq.com/gateway-operator
  parametersRef:
    group: gateway-operator.konghq.com
    kind: GatewayConfiguration
    name: kong
    namespace: default
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: kong2
  namespace: default
spec:
  gatewayClassName: kong
  listeners:
  - name: http
    protocol: HTTP
    port: 80
    allowedRoutes:
      kinds:
      - kind: HTTPRoute
      namespaces:
        from: Selector
        selector:
          matchLabels:
            kubernetes.io/metadata.name: kong-system # this should match 0 routes as the only one deployed here is in 'default' namespace

Kong Ingress Controller version

3.1.5

Kubernetes version

No response

Proposed solution

  • take listeners allowedRoutes into account when clearing the HTTPRoute status
  • add filtering for HTTPRoutes in HTTPRoute controller so that on updates, HTTPRoute is reconciled when
    • it was created attached to a Gateway that is being reconciled
    • it becomes attached to a Gateway that is being reconciled
    • it becomes detached from a Gateway that is being reconciled: this should cause the removal of the route from dataplane's configuration