aws/amazon-vpc-cni-k8s

Add Windows IPAM options as Environment Variables

tip-dteller opened this issue · 8 comments

tip-dteller commented

What would you like to be added:
Would like to add the following as environment variables.

  enable-windows-ipam: "false"
  enable-network-policy-controller: "false"
  enable-windows-prefix-delegation: "false"

Why is this needed:
When running an EKS hybrid cluster - linux and windows. it is required per AWS documentation to add
'enable-windows-ipam=true' as a VPC-CNI ConfigMap, making IPs available for allocation to Windows Pods.

There's no easy to add this when building a cluster using Terraform or any other IaC tool.
The steps required are:

  1. Create EKS Cluster and 1 Node Group.
  2. Wait for completion.
  3. Add ConfigMap using TF or manually.
  4. Add Windows Node Group - so it can work off the bat.

It would be far simpler to apply already in EKS Addon section of Terraform:

module "eks" {
  source  = "terraform-aws-modules/eks/aws"

  # Truncated for brevity
  ...

  cluster_addons = {
    vpc-cni = {
      before_compute = true
      most_recent    = true # To ensure access to the latest settings provided
      configuration_values = jsonencode({
        env = {
          ENABLE_PREFIX_DELEGATION = "true"
          WARM_PREFIX_TARGET       = "1"
          ENABLE_WINDOWS_IPAM = "true"
        }
      })
    }
  }

Env Vars should be loaded from the aws-vpc-cni\main.go - main.go

Subsequent logic should be added in relevant functions.

jdn5126 commented

@tip-dteller this configurability is already added in the EKS addon configuration for VPC CNI v1.16.0-eksbuild.1..

VPC CNI v1.16.0 is already available on GitHub: https://github.com/aws/amazon-vpc-cni-k8s/releases/tag/v1.16.0, and those fields are configurable in the ConfigMap via the helm chart: https://github.com/aws/amazon-vpc-cni-k8s/blob/v1.16.0/charts/aws-vpc-cni/templates/configmap.yaml#L20

For the EKS addon API you linked above, using configuration_values, you will be able to pass those fields from the top level, i.e.:

configuration_values = jsonencode({
  enableWindowsIpam = "true"
  enableWindowsPrefixDelegation = "true"
  warmWindowsIPTarget = 5
})

The EKS addon for v1.16.0-eksbuild.1 is currently in the pipeline, but is blocked by the holiday block days. It should be available across all regions by 12/28

  • 👍1
tip-dteller commented

@jdn5126 - Thanks for the reply,
so it will be available in 1.16, meaning an upgrade is required to use this via IaC solutions.

Also, why in the helm chart the ConfigMap is created as an objected but isnt attached to the daemonset? is that by design?
what reads the configmap?

jdn5126 commented

@jdn5126 - Thanks for the reply, so it will be available in 1.16, meaning an upgrade is required to use this via IaC solutions.

Correct

Also, why in the helm chart the ConfigMap is created as an objected but isnt attached to the daemonset? is that by design? what reads the configmap?

The ConfigMap is consumed by the controller: https://github.com/aws/amazon-vpc-resource-controller-k8s. In Windows deployments, the AWS VPC CNI is not used, as the controller handles all of the pod networking setup. These Windows variables are present in this helm chart only to make it easier for customers to use the same helm chart across Linux and Windows environments. This is where the AWS VPC CNI is prevented from being scheduled on Windows nodes: https://github.com/aws/amazon-vpc-cni-k8s/blob/master/config/master/aws-k8s-cni.yaml#L561

tip-dteller commented

Hmm I suppose it should be documented under some section in the documentation, since this part confused me, if you hadnt mentioned it, i'd be chasing a tail, trying to figure out what consumes those variables and how.

github-actions commented

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

VLZZZ commented

I just want to add more context on why it might be misleading when using Helm chart.

During the upgrade to v1.16.0 we encountered with the following change in our dry-run job:
image

And it took some time to understand how it will (or not) affect our configuration.
Because the env var WARM_PREFIX_TARGET (for linux) and config map entry warm-prefix-target (for windows) are almost identical.

Moreover when reading documentation you might think that - "Ok, Linux is using env vars for configuration, while Windows using config file entries", but as far as I remember from this discussion on IPv6 Disablement config file is just a config file that's not tied to the target OS and you may have some common configuration here.

For me it would make more sense to have all windows-related setting to be configurable not only in the atomic way, but also to have main switch for the Helm chart, like windows.enabled: false (just to reflect my idea) to have full control over configuration (if I don't have any Windows/Linux hosts, why do I need some configuration, which is controller in two different ways) and avoid misleading changes like this.

jdn5126 commented

@VLZZZ yeah, the naming for these ConfigMap variables was a big miss. The names are fixed because that is what the VPC Resource Controller looks at for Windows Prefix Delegation (https://github.com/aws/amazon-vpc-resource-controller-k8s/blob/master/pkg/config/type.go#L76). We need to update the names in a future release.

  • 👍1