[aws-load-balancer-controller] repeatedly attaching/detaching network interfaces

[razorsk8jz]

Describe the bug
We just started noticing alerts coming in that network interfaces were getting attached and detached repeatedly 4 days ago. This is happening in multiple environments using the chart version 1.7.1. The logs for the controller appear to show {"level":"info","ts":"2024-04-19T16:19:11Z","logger":"controllers.ingress","msg":"successfully deployed model","ingressGroup":"monitoring/kube-prometheus-stack-grafana"} {"level":"info","ts":"2024-04-19T16:19:12Z","logger":"controllers.ingress","msg":"Auto Create SG","LB SGs":[{"$ref":"#/resources/AWS::EC2::SecurityGroup/ManagedLBSecurityGroup/status/groupID"},"sg-03cd...."],"backend SG":"sg-03cd......"}

Any insights on why this attach/detach just started happening multiple times per day and is this normal?

Steps to reproduce
Not sure whats causing so not sure how to reproduce

Expected outcome
Its expected that load balancers are only updated when the model changes or finds something wrong with the provisioned lb

Environment

• Chart name: aws-load-balancer-controller
• Chart version: 1..7.1
• Kubernetes version: 1.28.5
• Using EKS (yes/no), if so version? N/A Kops

Additional Context:
here is an aws event we received

{ "awsRegion": "us-east-1", "eventCategory": "Management", "eventID": "....", "eventName": "DetachNetworkInterface", "eventSource": "ec2.amazonaws.com", "eventTime": "2024-04-19T15:06:56Z", "eventType": "AwsApiCall", "eventVersion": "1.09", "managementEvent": true, "readOnly": false, "recipientAccountId": "....", "requestID": "....", "requestParameters": { "attachmentId": "eni-attach-....", "force": true }, "responseElements": { "_return": true, "requestId": "...." }, "sharedEventID": "....", "sourceIPAddress": "elasticloadbalancing.amazonaws.com", "userAgent": "elasticloadbalancing.amazonaws.com", "userIdentity": { "invokedBy": "elasticloadbalancing.amazonaws.com", "type": "AWSService" } }