terraform-aws-modules/terraform-aws-eks

min_size & desired_size

dmitry-mightydevops opened this issue · 6 comments

dmitry-mightydevops commented

Description

eks_managed_node_groups = {
  ops-v1 = {
    min_size       = 1
    max_size       = 5
    desired_size   = 1
    ...
    instance_types = ["m5.large"]
  }

the group is created. I change the desired_size 1 -> 2, run apply and nothing. It doesn't detect the change because of the following code for the node group

lifecycle {
    create_before_destroy = true
    ignore_changes = [
      scaling_config[0].desired_size,
    ]
  }

I then try to do

eks_managed_node_groups = {
  ops-v1 = {
    min_size       = 2
    max_size       = 5
    desired_size   = 2
    ...
    instance_types = ["m5.large"]
  }

and apply fails with the following error:

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # module.eks_cluster.module.eks_cluster.module.eks_managed_node_group["ops-v1"].aws_eks_node_group.this[0] will be updated in-place
  ~ resource "aws_eks_node_group" "this" {
        id              = "project-prod-eks:prod-ops-v1"
        tags            = {
            "Name"                                = "prod-ops-v1"
            "created"                             = "4/18/2022"
            "environment"                         = "prod"
            "updated"                             = "4/18/2022"

        }
        # (15 unchanged attributes hidden)


      ~ scaling_config {
          ~ min_size     = 1 -> 2
            # (2 unchanged attributes hidden)
        }


        # (3 unchanged blocks hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

Do you want to perform these actions in workspace "prod"?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

module.eks_cluster.module.eks_cluster.module.eks_managed_node_group["ops-v1"].aws_eks_node_group.this[0]: Modifying... [id=project-prod-eks:prod-ops-v1]
╷
│ Error: error updating EKS Node Group (project-prod-eks:prod-ops-v1) config: InvalidParameterException: Minimum capacity 2 can't be greater than desired size 1
│ {
│   RespMetadata: {
│     StatusCode: 400,
│     RequestID: "21006d45-9a28-40ec-8d06-0205d6abc95d"
│   },
│   ClusterName: "project-prod-eks",
│   Message_: "Minimum capacity 2 can't be greater than desired size 1",
│   NodegroupName: "prod-ops-v1"
│ }
│ 
│   with module.eks_cluster.module.eks_cluster.module.eks_managed_node_group["ops-v1"].aws_eks_node_group.this[0],
│   on .terraform/modules/eks_cluster.eks_cluster/modules/eks-managed-node-group/main.tf line 272, in resource "aws_eks_node_group" "this":
│  272: resource "aws_eks_node_group" "this" {
│ 
╵
make: *** [Makefile:38: apply] Error 1

So the only way to overcome it is

  • modify ASG manually in GUI or via AWS cli
  • create a new "node-group" with different basic settings.

Is this considered a bug?

  • ✋ I have searched the open/closed issues and my issue is not listed.

Versions

  • Module version [Required]:

  • Terraform version: 1.1.8

  • Provider version(s): 18.20.2

bryantbiggs commented

hi @dmitry-mightydevops - this is not a bug and is the desired effect. Autoscaling of nodes via cluster autoscaler or Karpenter is the preferred method by many, and unfortunately Terraform does not allow for parameterizing ignore_changes at this time so this is the default effect

dmitry-mightydevops commented

Sounds good - thanks @bryantbiggs that's what I thought.

  • 👍1
NickLarsenNZ commented

I'm getting this too (on a cluster I scaled down manually via the console).

@bryantbiggs I think this should be reopened.

Perhaps the min_size, max_size, and desired_size values should change in the order depending on which is currently greater or less than others (to avoid breaking the constraint).

More info

My terraform plan shows the changes made outside of what the state knows (I had scaled it to 1):

      ~ scaling_config {
          ~ desired_size = 3 -> 1
          ~ min_size     = 3 -> 1
            # (1 unchanged attribute hidden)
        }

I then see that terraform will increase the min_size:

Note: I just noticed that the desired_size change doesn't appear below. I have checked to see if we (or the module we use) is ignoring the changes, but I can't see any occurences of that. Maybe this is a different issue

      ~ scaling_config {
          ~ min_size     = 1 -> 3
            # (2 unchanged attributes hidden)
        }

Then during apply, it fails with:

│ Error: error updating EKS Node Group (multitenant-capetown-prod:initial-20220901104640951800000008) config: InvalidParameterException: Minimum capacity 3 can't be greater than desired size 1
│ {
│   RespMetadata: {
│     StatusCode: 400,
│     RequestID: "6a4f9ce0-8942-4b1b-b5db-880e95ad72c6"
│   },
│   ClusterName: "my-cluster",
│   Message_: "Minimum capacity 3 can't be greater than desired size 1",
│   NodegroupName: "initial-20220901104640951800000009"
│ }
│ 
│   with module.eks.module.eks_managed_node_group["initial"].aws_eks_node_group.this[0],
│   on .terraform/modules/eks/modules/eks-managed-node-group/main.tf line 272, in resource "aws_eks_node_group" "this":
│  272: resource "aws_eks_node_group" "this" {
  • 👍1
dejwsz commented

Regarding the comment: #2030 (comment) I suspect it will be possible to handle this properly after this is done: hashicorp/terraform#27360, hashicorp/terraform#3116

github-actions commented

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.