konveyor/operator

EKS with ALB: Changes required to work with Amazon Load Balancer ingress controller

Closed this issue · 1 comments

jwmatthews commented

Related #167 and #170

On an AWS EKS cluster with the AWS LoadBalancer configured as Ingress, the operator is unable to create the Ingress resource

due to

spec:
  ingressClassName: nginx

from operator logs

--------------------------- Ansible Task StdOut -------------------------------

TASK [tackle : Setup UI Ingress] ***********************************************
task path: /opt/ansible/roles/tackle/tasks/main.yml:369

-------------------------------------------------------------------------------
{"level":"info","ts":1680981644.5812697,"logger":"proxy","msg":"Cache miss: networking.k8s.io/v1, Kind=Ingress, konveyor-tackle/tackle"}
{"level":"info","ts":1680981644.5886374,"logger":"proxy","msg":"Injecting owner reference"}
{"level":"info","ts":1680981644.5889103,"logger":"proxy","msg":"Watching child resource","kind":"networking.k8s.io/v1, Kind=Ingress","enqueue_kind":"tackle.konveyor.io/v1alpha1, Kind=Tackle"}
{"level":"info","ts":1680981644.588938,"msg":"Starting EventSource","controller":"tackle-controller","source":"kind source: *unstructured.Unstructured"}
{"level":"error","ts":1680981644.856265,"logger":"logging_event_handler","msg":"","name":"tackle","namespace":"konveyor-tackle","gvk":"tackle.konveyor.io/v1alpha1, Kind=Tackle","event_type":"runner_on_failed","job":"1812945292459884819","EventData.Task":"Setup UI Ingress","EventData.TaskArgs":"","EventData.FailedTaskPath":"/opt/ansible/roles/tackle/tasks/main.yml:369","error":"[playbook task failed]"}

--------------------------- Ansible Task StdOut -------------------------------

 TASK [Setup UI Ingress] ******************************** 
fatal: [localhost]: FAILED! => {"changed": false, "error": 403, "msg": "Ingress tackle: Failed to create object: b'{\"kind\":\"Status\",\"apiVersion\":\"v1\",\"metadata\":{},\"status\":\"Failure\",\"message\":\"admission webhook \\\\\"vingress.elbv2.k8s.aws\\\\\" denied the request: invalid ingress class: IngressClass.networking.k8s.io \\\\\"nginx\\\\\" not found\",\"reason\":\"invalid ingress class: IngressClass.networking.k8s.io \\\\\"nginx\\\\\" not found\",\"code\":403}\\n'", "reason": "Forbidden", "status": 403}
jwmatthews commented

I collected a list of changes needed here:

  1. Stop the operator from reconciling so we can make manual tweaks
kubectl -n konveyor-tackle scale deployment/tackle-operator --replicas=0
  1. Change Konveyor's Ingress 'tackle' to
spec:
  ingressClassName: alb
  1. Either add the annotation alb.ingress.kubernetes.io/target-type: IP or change the 'tackle-ui' service from 'ClusterIP' to 'NodePort'
  • Background:
    • I saw the below error and no external address in the Ingress I could use. I believe it was because we are using 'ClusterIP' and not 'NodePort'. Docs I saw on ALB gave me the impression it prefers to work with Services of type 'NodePort'.
      • With no external address listed in the Ingress, look at the ingress controller logs 
        kubectl logs -f -n kube-system -l app.kubernetes.io/name=aws-load-balancer-controller

Log output from aws-load-balancer-controller

{"level":"info","ts":1681061324.1948602,"logger":"controllers.ingress","msg":"Auto Create SG","LB SGs":[{"$ref":"#/resources/AWS::EC2::SecurityGroup/ManagedLBSecurityGroup/status/groupID"},"sg-02294ae7a644d73d0"],"backend SG":"sg-02294ae7a644d73d0"}
{"level":"info","ts":1681061324.1951888,"logger":"controllers.ingress","msg":"successfully built model","model":"{\"id\":\"konveyor-tackle/tackle\",\"resources\":{\"AWS::EC2::SecurityGroup\":{\"ManagedLBSecurityGroup\":{\"spec\":{\"groupName\":\"k8s-konveyor-tackle-70e9f7ad01\",\"description\":\"[k8s] Managed SecurityGroup for LoadBalancer\",\"ingress\":[{\"ipProtocol\":\"tcp\",\"fromPort\":80,\"toPort\":80,\"ipRanges\":[{\"cidrIP\":\"0.0.0.0/0\"}]}]}}},\"AWS::ElasticLoadBalancingV2::Listener\":{\"80\":{\"spec\":{\"loadBalancerARN\":{\"$ref\":\"#/resources/AWS::ElasticLoadBalancingV2::LoadBalancer/LoadBalancer/status/loadBalancerARN\"},\"port\":80,\"protocol\":\"HTTP\",\"defaultActions\":[{\"type\":\"fixed-response\",\"fixedResponseConfig\":{\"contentType\":\"text/plain\",\"statusCode\":\"404\"}}]}}},\"AWS::ElasticLoadBalancingV2::ListenerRule\":{\"80:1\":{\"spec\":{\"listenerARN\":{\"$ref\":\"#/resources/AWS::ElasticLoadBalancingV2::Listener/80/status/listenerARN\"},\"priority\":1,\"actions\":[{\"type\":\"forward\",\"forwardConfig\":{\"targetGroups\":[{\"targetGroupARN\":{\"$ref\":\"#/resources/AWS::ElasticLoadBalancingV2::TargetGroup/konveyor-tackle/tackle-tackle-ui:8080/status/targetGroupARN\"}}]}}],\"conditions\":[{\"field\":\"path-pattern\",\"pathPatternConfig\":{\"values\":[\"/\"]}}]}}},\"AWS::ElasticLoadBalancingV2::LoadBalancer\":{\"LoadBalancer\":{\"spec\":{\"name\":\"k8s-konveyor-tackle-1b35dd37b3\",\"type\":\"application\",\"scheme\":\"internal\",\"ipAddressType\":\"ipv4\",\"subnetMapping\":[{\"subnetID\":\"subnet-0071c04883eb68311\"},{\"subnetID\":\"subnet-08fe6c3d940d8f155\"}],\"securityGroups\":[{\"$ref\":\"#/resources/AWS::EC2::SecurityGroup/ManagedLBSecurityGroup/status/groupID\"},\"sg-02294ae7a644d73d0\"]}}},\"AWS::ElasticLoadBalancingV2::TargetGroup\":{\"konveyor-tackle/tackle-tackle-ui:8080\":{\"spec\":{\"name\":\"k8s-konveyor-tackleui-01ccea48c3\",\"targetType\":\"instance\",\"port\":0,\"protocol\":\"HTTP\",\"protocolVersion\":\"HTTP1\",\"ipAddressType\":\"ipv4\",\"healthCheckConfig\":{\"port\":\"traffic-port\",\"protocol\":\"HTTP\",\"path\":\"/\",\"matcher\":{\"httpCode\":\"200\"},\"intervalSeconds\":15,\"timeoutSeconds\":5,\"healthyThresholdCount\":2,\"unhealthyThresholdCount\":2}}}},\"K8S::ElasticLoadBalancingV2::TargetGroupBinding\":{\"konveyor-tackle/tackle-tackle-ui:8080\":{\"spec\":{\"template\":{\"metadata\":{\"name\":\"k8s-konveyor-tackleui-01ccea48c3\",\"namespace\":\"konveyor-tackle\",\"creationTimestamp\":null},\"spec\":{\"targetGroupARN\":{\"$ref\":\"#/resources/AWS::ElasticLoadBalancingV2::TargetGroup/konveyor-tackle/tackle-tackle-ui:8080/status/targetGroupARN\"},\"targetType\":\"instance\",\"serviceRef\":{\"name\":\"tackle-ui\",\"port\":8080},\"networking\":{\"ingress\":[{\"from\":[{\"securityGroup\":{\"groupID\":\"sg-02294ae7a644d73d0\"}}],\"ports\":[{\"protocol\":\"TCP\",\"port\":0}]}]},\"ipAddressType\":\"ipv4\"}}}}}}}"}
{"level":"info","ts":1681061324.9594808,"logger":"controllers.ingress","msg":"creating targetGroup","stackID":"konveyor-tackle/tackle","resourceID":"konveyor-tackle/tackle-tackle-ui:8080"}
{"level":"error","ts":1681061324.959598,"logger":"controller.ingress","msg":"Reconciler error","name":"tackle","namespace":"konveyor-tackle","error":"InvalidParameter: 1 validation error(s) found.\n- minimum field value of 1, CreateTargetGroupInput.Port.\n"}
  • To fix above I changed the referring Service from ClusterIP to NodePort
  1. Then I received an address to use, but it was prefixed with "internal-*" for the ALB. I couldn't reach the address as my browser was out of the cluster.
  • I was able to change this to an internet facing address with the below annotation
  • Add an annotation: 
    alb.ingress.kubernetes.io/scheme: internet-facing
  1. Change the ALB config from pathType: ImplementationSpecific to pathType: Prefix
  • Background: Next issue...I can reach the Ingress address and I fetch index.html and installHooks.js, but I can't fetch app.css or app.bundle.js, I received 404's for them. The page is not rendered, I see just a white screen and there are not errors in the pod running tackle-ui.
  • To debug further, I manually ran a port-forward to the tackle-ui service to confirm it was working, and from it I was able to render the webui as expected. Now I've confirmed the UI service is working and the issue is in the ALB. 
    kubectl port-forward svc/tackle-ui 7080:8080 -n konveyor-tackle
  • Changing from pathType: ImplementationSpecific to pathType: Prefix resolved the issue

Ingress was setup as below:

  spec:
    ingressClassName: alb
    rules:
    - http:
        paths:
        - backend:
            service:
              name: tackle-ui
              port:
                number: 8080
          path: /
          pathType: ImplementationSpecific
    tls:
    - {}

Resolved by changing to 'pathType: Prefix'

  spec:
    ingressClassName: alb
    rules:
    - http:
        paths:
        - backend:
            service:
              name: tackle-ui
              port:
                number: 8080
          path: /
          pathType: Prefix
    tls:
    - {}

With all of the above changes I was able to access the tackle UI from the ingress address on an EKS Cluster with ALB ingress