Warning FailedScheduling <invalid> (x7 over 26s) default-scheduler 0/5 nodes are available: 5 Insufficient cpu

[nkhine]

Hello,
I have setup a k8s cluster with the following:

$ kops validate cluster
Using cluster from kubectl context: es.k8s.local

Validating cluster es.k8s.local

INSTANCE GROUPS
NAME                    ROLE    MACHINETYPE     MIN     MAX     SUBNETS
bastions                Bastion t2.micro        1       1       utility-eu-west-1a
data                    Node    t2.medium       2       2       eu-west-1a
master-eu-west-1a       Master  m3.medium       1       1       eu-west-1a
nodes                   Node    t2.medium       2       2       eu-west-1a

NODE STATUS
NAME                                            ROLE    READY
ip-172-20-39-51.eu-west-1.compute.internal      node    True
ip-172-20-40-39.eu-west-1.compute.internal      node    True
ip-172-20-43-105.eu-west-1.compute.internal     node    True
ip-172-20-44-5.eu-west-1.compute.internal       node    True
ip-172-20-62-224.eu-west-1.compute.internal     master  True

Your cluster es.k8s.local is ready

so I have one master, 2 nodes and 2 data nodes

when I try to add the es-client, this fails, with the following error:

$ kubectl describe po es-client-7b59bd6d64-vmvv5
Name:           es-client-7b59bd6d64-vmvv5
Namespace:      default
Node:           <none>
Labels:         component=elasticsearch
                pod-template-hash=3615682820
                role=client
Annotations:    kubernetes.io/limit-ranger=LimitRanger plugin set: cpu request for init container init-sysctl
Status:         Pending
IP:
Controlled By:  ReplicaSet/es-client-7b59bd6d64
Init Containers:
  init-sysctl:
    Image:  busybox:1.27.2
    Port:   <none>
    Command:
      sysctl
      -w
      vm.max_map_count=262144
    Requests:
      cpu:        100m
    Environment:  <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-nb2zw (ro)
Containers:
  es-client:
    Image:  quay.io/pires/docker-elasticsearch-kubernetes:6.2.2_1
    Ports:  9200/TCP, 9300/TCP
    Limits:
      cpu:  2
    Requests:
      cpu:      2
    Liveness:   tcp-socket :transport delay=0s timeout=1s period=10s #success=1 #failure=3
    Readiness:  http-get http://:http/_cluster/health delay=20s timeout=5s period=10s #success=1 #failure=3
    Environment:
      NAMESPACE:     default (v1:metadata.namespace)
      NODE_NAME:     es-client-7b59bd6d64-vmvv5 (v1:metadata.name)
      CLUSTER_NAME:  es
      NODE_MASTER:   false
      NODE_DATA:     false
      HTTP_ENABLE:   true
      ES_JAVA_OPTS:  -Xms256m -Xmx256m
      NETWORK_HOST:  _site_,_lo_
      PROCESSORS:    2 (limits.cpu)
    Mounts:
      /data from storage (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-nb2zw (ro)
Conditions:
  Type           Status
  PodScheduled   False
Volumes:
  storage:
    Type:    EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
  default-token-nb2zw:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-nb2zw
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s

Events:
  Type     Reason            Age                      From               Message
  ----     ------            ----                     ----               -------
  Warning  FailedScheduling  <invalid> (x13 over 2m)  default-scheduler  0/5 nodes are available: 5 Insufficient cpu.

what am i missing?

[pires]

0/5 nodes are available: 5 Insufficient cpu. you don't have enough CPU available in your Kubernetes nodes.

[nkhine]

@pires what ec2 node size would you recommend for a test setup?

[pires]

Honestly, I don't have an answer for that :(

[ameukam]

Hi @nkhine, I think you should start with the t2.large for the nodes and scale it out depending of your consumption.

[yangwenmai]

You can exec cat /proc/cpuinfo to check cpu.
May be your machine's Insufficient cpu, Because :

es-master.yaml and es-client.yaml have config replicas: 3 and replicas: 2, It's 5 CPU.

You can set replicas: 1, It's ok.