Remove AmazonEKS_CNI_Policy from DefaultManagedPolicies

[backjo]

AmazonEKS_CNI_Policy is currently part of DefaultManagedPolicies, which are automatically added to the roles created and associated with nodes provisioned by the EKS provisioner.

In the AWS docs, it is recommended to remove it and use IAM Roles for Service Accounts instead.
We should remove it from the defaults or offer a way for users to opt-out of attaching it.

[eytan-avisror]

We will need to keep involvement here minimal, but can allow control to adding policy to controller-owned role by using an annotation like instancemgr.keikoproj.io/use-irsa="true"

• if annotation is true, and role is owned by controller, don't add AmazonEKS_CNI_Policy to role / remove it.
• this is only relevant if IAM role is owned by the controller - if you bring our own role (by providing roleName / instanceProfileName - there will be no effect and you are responsible 100% for migrating to IRSA, as well as burden of configuring OIDC correctly should be on the user, controller should not try to configure OIDC for the user.

Possible Migration Path for using controller-owned roles

• User makes sure OIDC provider is created/associated for the cluster:
  https://docs.aws.amazon.com/eks/latest/userguide/enable-iam-roles-for-service-accounts.html

• User makes sure trust relationship exists between OIDC Provider/aws-node by creating a new role containing the AmazonEKS_CNI_Policy as per https://docs.aws.amazon.com/eks/latest/userguide/cni-iam-role.html

• User annotates aws-node to so that it uses the role as per https://docs.aws.amazon.com/eks/latest/userguide/cni-iam-role.html

• User adds the use-irsa=true annotation on the IG to remove AmazonEKS_CNI_Policy from managed role

• If user owns the role outside of controller, he should remove it manually (in this case, migration is manual as we do not add this policy).

WDYT?

[backjo]

Yep, that seems to be along the same lines of what I was thinking. A new annotation seems like the cleanest way.