cluster-autoscaler: Wrong configuration for priority expander

[elliotdobson]

/kind bug

1. What kops version are you running? The command kops version, will display
this information.
Client version: 1.26.6 (git-v1.26.6)

2. What Kubernetes version are you running? kubectl version will print the
version if a cluster is running or provide the Kubernetes version specified as
a kops flag.
Server Version: v1.26.15

3. What cloud provider are you using?
AWS

4. What commands did you run? What is the simplest way to reproduce this issue?
Configure the cluster-autoscaler managed-addon using expander: priority & instancegroups with autoscale: true & autoscalePriority set.

5. What happened after the commands executed?
Cluster-autoscaler fails to use the cluster-autoscaler-priority-expander configMap that is created due to this error:

$ kubectl describe configmap cluster-autoscaler-priority-expander
Name:         cluster-autoscaler-priority-expander
Namespace:    kube-system
Labels:       addon.kops.k8s.io/name=cluster-autoscaler.addons.k8s.io
              app.kubernetes.io/managed-by=kops
              app.kubernetes.io/name=cluster-autoscaler
              k8s-addon=cluster-autoscaler.addons.k8s.io
              k8s-app=cluster-autoscaler
Annotations:  <none>

Data
====
priorities:
----
"0":
- nodes-a.[cluster_name].[domain]
- nodes-b.[cluster_name].[domain]
"10":
- nodes-cifs.[cluster_name].[domain]
"20":
- nodes-a-ondemand.[cluster_name].[domain]
- nodes-b-ondemand.[cluster_name].[domain]
"30":
- nodes-b-spot.[cluster_name].[domain]
- nodes-a-spot.[cluster_name].[domain]

BinaryData
====

Events:
  Type     Reason                    Age   From                Message
  ----     ------                    ----  ----                -------
  Warning  PriorityConfigMapInvalid  17m   cluster-autoscaler  Wrong configuration for priority expander: Can't parse YAML with priorities in the configmap: yaml: unmarshal errors:
  line 1: cannot unmarshal !!str `0` into int
  line 5: cannot unmarshal !!str `10` into int
  line 7: cannot unmarshal !!str `20` into int
  line 10: cannot unmarshal !!str `30` into int. Ignoring update.

6. What did you expect to happen?
The cluster-autoscaler-priority-expander configMap to be created with the correct formatting as per docs & example.

7. Please provide your cluster manifest. Execute
kops get --name my.example.com -o yaml to display your cluster manifest.
You may want to remove your cluster name and other sensitive information.

apiVersion: kops.k8s.io/v1alpha2
kind: Cluster
...
spec:
...
  cloudProvider: aws
  clusterAutoscaler:
    awsUseStaticInstanceList: false
    balanceSimilarNodeGroups: true
    cordonNodeBeforeTerminating: false
    cpuRequest: 100m
    enabled: true
    expander: priority
    maxNodeProvisionTime: 10m0s
    memoryRequest: 384Mi
    newPodScaleUpDelay: 30s
    scaleDownDelayAfterAdd: 10m0s
    scaleDownUnneededTime: 10m0s
    scaleDownUnreadyTime: 20m0s
    scaleDownUtilizationThreshold: "0.5"
    skipNodesWithLocalStorage: false
    skipNodesWithSystemPods: false
...

apiVersion: kops.k8s.io/v1alpha2
kind: InstanceGroup
...
spec:
  autoscale: true
  autoscalePriority: 30
...

8. Please run the commands with most verbose logging by adding the -v 10 flag.
Paste the logs into this report, or in a gist and provide the gist link here.

9. Anything else do we need to know?
I also tried using the customPriorityExpanderConfig in the kOps cluster spec but it produced the same cluster-autoscaler-priority-expander configMap.

    customPriorityExpanderConfig:
      10:
      - .*
      20:
      - .*nodes-a-ondemand.*
      - .*nodes-b-ondemand.*
      30:
      - .*nodes-a-spot.*
      - .*nodes-b-spot.*

I then used createPriorityExpanderConfig: false in the kOps cluster spec and created the cluster-autoscaler-priority-expander configMap manually using the yaml below, and that resolved my issues with cluster-autoscaler able to read the priority list correctly.

apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-autoscaler-priority-expander
  namespace: kube-system
  labels:
    app: cluster-autoscaler
data:
  priorities: |-
    10:
      - ".*"
    20:
      - ".*nodes-a-ondemand.*"
      - ".*nodes-b-ondemand.*"
    30:
      - ".*nodes-a-spot.*"
      - ".*nodes-b-spot.*"