aws/karpenter-provider-aws

Kubelet configuration not applied after v1 upgrade

InsomniaCoder opened this issue · 5 comments

InsomniaCoder commented

Description

Observed Behavior:

After upgrading to v1 from 0.37.0, I've noticed that our kubelet configuration specified in the NodePools is not applied anymore.

These are some of the example:

kubelet:
    maxPods: 110
    podsPerCore: 6
    evictionHard:
      memory.available: 300Mi
      nodefs.available: 10%
      nodefs.inodesFree: 5%
      imagefs.inodesFree: 5%
    evictionSoft:
      memory.available: 500Mi
      nodefs.available: 15%
    evictionSoftGracePeriod:
      memory.available: 1m0s
      nodefs.available: 1m0s

in the nodepool of 0.37.0 when inspecting kubelet's log in the machine I'd see

--eviction-hard="imagefs.inodesFree<5%,memory.available<300Mi,nodefs.available<10%,nodefs.inodesFree<5%"
FLAG: --eviction-max-pod-grace-period="0"
FLAG: --eviction-minimum-reclaim=""
FLAG: --eviction-pressure-transition-period="5m0s"
FLAG: --eviction-soft="memory.available<500Mi,nodefs.available<15%"
FLAG: --eviction-soft-grace-period="memory.available=1m0s,nodefs.available=1m0s"

However, after v1 upgrade, they are gone.

I do see conversion annotation added to the nodepool

compatibility.karpenter.sh/v1beta1-kubelet-conversion:
                {"maxPods":110,"podsPerCore":6,"evictionHard":{"imagefs.inodesFree":"5%","memory.available":"300Mi","nodefs.available":"10%","nodefs.inode...
              compatibility.karpenter.sh/v1beta1-nodeclass-reference: {"name":"schip-default"}

Expected Behavior:

Once upgraded, you won't need to roll your nodes to be compatible with v1.1, except if you have multiple NodePools with different kubelets that are referencing the same EC2NodeClass. Karpenter has moved the kubelet to the EC2NodeClass in v1. NodePools with different kubelet referencing the same EC2NodeClass will be compatible with v1.0, but will not be in v1.1.

Using different kubelet config per NodePool is still compatible in v1.

Reproduction Steps (Please include YAML):

  • create a nodepool with custom kubelet configuration on Karpenter 0.37.0
  • Upgrade Karpenter from 0.37.0 to 1.0.1
  • Check kubelet configuration of a new node

Versions:

  • Chart Version: 1.0.1
  • Kubernetes Version (kubectl version): 1.28
  • Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
  • Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment
engedaam commented

@InsomniaCoder From my testing and initial code review, this is a bug we need to fix:

  • ❤️1
jonathan-innis commented

There's a change opened in the upstream repo with the fix: kubernetes-sigs/karpenter#1667. We should be raising a patch for this issue, but, in the interim, here's a snapshot that you can install in dev if you'd like to test the fix.

aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin 021119463062.dkr.ecr.us-east-1.amazonaws.com

helm upgrade --install karpenter oci://021119463062.dkr.ecr.us-east-1.amazonaws.com/karpenter/snapshot/karpenter --version "0-2f61ca341eaf5f220a0e70ee12c5d6d6c6c00438" --namespace "kube-system" --create-namespace \
  --set "settings.clusterName=${CLUSTER_NAME}" \
  --set "settings.interruptionQueue=${CLUSTER_NAME}" \
  --set controller.resources.requests.cpu=1 \
  --set controller.resources.requests.memory=1Gi \
  --set controller.resources.limits.cpu=1 \
  --set controller.resources.limits.memory=1Gi \
  --wait

(see #7013)

  • ❤️2
InsomniaCoder commented

There's a change opened in the upstream repo with the fix: kubernetes-sigs/karpenter#1667. We should be raising a patch for this issue, but, in the interim, here's a snapshot that you can install in dev if you'd like to test the fix.

aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin 021119463062.dkr.ecr.us-east-1.amazonaws.com

helm upgrade --install karpenter oci://021119463062.dkr.ecr.us-east-1.amazonaws.com/karpenter/snapshot/karpenter --version "0-2f61ca341eaf5f220a0e70ee12c5d6d6c6c00438" --namespace "kube-system" --create-namespace \
  --set "settings.clusterName=${CLUSTER_NAME}" \
  --set "settings.interruptionQueue=${CLUSTER_NAME}" \
  --set controller.resources.requests.cpu=1 \
  --set controller.resources.requests.memory=1Gi \
  --set controller.resources.limits.cpu=1 \
  --set controller.resources.limits.memory=1Gi \
  --wait

(see #7013)

thanks @jonathan-innis for sharing the fix. for curiosity, this change will make sure the kubelet annotation got propagated to the nodeclaims. Can you point me to where this annotation is then used/extracted to apply the kubelet configuration?

jonathan-innis commented

Can you point me to where this annotation is then used/extracted to apply the kubelet configuration

Sure -- it's used here: https://github.com/aws/karpenter-provider-aws/blob/release-v1.0.x/pkg/providers/amifamily/resolver.go#L210

mdesouky commented

So the kubelet configuration has now been moved to the nodeClass instead of the nodePools and when updated on EKS clusters, it takes effect on the nodeClass object itself but doesn't cause drift or Instance recycling.
We're on v1.0 still

  • 👍1