kvaps/kubectl-node-shell

Unable to spin shell on a node with taints

amnk opened this issue · 6 comments

amnk commented

First of all - thank you very much for the project. It is elegant and useful!

It saved my day today, but only partially - because it does not support nodes with taints. Would be great to see it being able to spin shells on any type of node.

kvaps commented

Hi, are you sure that this is not working with taints?

When you creating pod with the spec.nodeName specified it will always be run on node regardless of it's taints.

amnk commented

@kvaps I have two nodepools at the moment. Tool works perfectly fine on a nodepool without taints, and fails on a pool with taints (at least that is the only different that I can spot). It fails like this:

❯ k node-shell xxxxx
spawning "nsenter-p4kc5e" on "xxxx"
No resources found
Error from server (NotFound): pods "nsenter-p4kc5e" not found

If I can provide any other info - let me know.

kvaps commented

OK, please provide me your node yaml:

k get node <node_name> -o yaml

and

k version

I'll try to reproduce this

amnk commented
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.4", GitCommit:"c96aede7b5205121079932896c4ad89bb93260af", GitTreeState:"clean", BuildDate:"2020-06-18T02:59:13Z", GoVersion:"go1.14.3", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.6", GitCommit:"dff82dc0de47299ab66c83c626e08b245ab19037", GitTreeState:"clean", BuildDate:"2020-07-15T16:51:04Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

and

apiVersion: v1
kind: Node
metadata:
  annotations:
    alpha.kubernetes.io/provided-node-ip: 10.136.206.175
    csi.volume.kubernetes.io/nodeid: '{"dobs.csi.digitalocean.com":"201815603"}'
    io.cilium.network.ipv4-cilium-host: 10.244.4.161
    io.cilium.network.ipv4-health-ip: 10.244.4.194
    io.cilium.network.ipv4-pod-cidr: 10.244.4.128/25
    node.alpha.kubernetes.io/ttl: "0"
    volumes.kubernetes.io/controller-managed-attach-detach: "true"
  creationTimestamp: "2020-07-28T20:59:46Z"
  labels:
    beta.kubernetes.io/arch: amd64
    beta.kubernetes.io/instance-type: s-4vcpu-8gb
    beta.kubernetes.io/os: linux
    cluster: python-apps
    doks.digitalocean.com/node-id: a8fe8fea-e6c6-4cd8-a35c-15f836448016
    doks.digitalocean.com/node-pool: python-apps
    doks.digitalocean.com/node-pool-id: 698d5eca-8ecd-4538-8167-bfc193711765
    doks.digitalocean.com/version: 1.18.6-do.0
    failure-domain.beta.kubernetes.io/region: nyc1
    kubernetes.io/arch: amd64
    kubernetes.io/hostname: python-apps-3re6j
    kubernetes.io/os: linux
    node.kubernetes.io/instance-type: s-4vcpu-8gb
    region: nyc1
    topology.kubernetes.io/region: nyc1
  managedFields:
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:labels:
          f:beta.kubernetes.io/instance-type: {}
          f:failure-domain.beta.kubernetes.io/region: {}
          f:node.kubernetes.io/instance-type: {}
          f:topology.kubernetes.io/region: {}
      f:status:
        f:addresses:
          k:{"type":"ExternalIP"}:
            .: {}
            f:address: {}
            f:type: {}
    manager: digitalocean-cloud-controller-manager
    operation: Update
    time: "2020-07-28T20:59:48Z"
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          f:node.alpha.kubernetes.io/ttl: {}
      f:spec:
        f:podCIDR: {}
        f:podCIDRs:
          .: {}
          v:"10.244.4.128/25": {}
    manager: kube-controller-manager
    operation: Update
    time: "2020-07-28T20:59:56Z"
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          f:io.cilium.network.ipv4-cilium-host: {}
          f:io.cilium.network.ipv4-health-ip: {}
          f:io.cilium.network.ipv4-pod-cidr: {}
      f:status:
        f:conditions:
          k:{"type":"NetworkUnavailable"}:
            .: {}
            f:lastHeartbeatTime: {}
            f:lastTransitionTime: {}
            f:message: {}
            f:reason: {}
            f:status: {}
            f:type: {}
    manager: cilium-agent
    operation: Update
    time: "2020-07-28T21:00:04Z"
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:labels:
          f:cluster: {}
      f:spec:
        f:taints: {}
    manager: kubectl
    operation: Update
    time: "2020-07-28T21:06:05Z"
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .: {}
          f:alpha.kubernetes.io/provided-node-ip: {}
          f:csi.volume.kubernetes.io/nodeid: {}
          f:volumes.kubernetes.io/controller-managed-attach-detach: {}
        f:labels:
          .: {}
          f:beta.kubernetes.io/arch: {}
          f:beta.kubernetes.io/os: {}
          f:doks.digitalocean.com/node-id: {}
          f:doks.digitalocean.com/node-pool: {}
          f:doks.digitalocean.com/node-pool-id: {}
          f:doks.digitalocean.com/version: {}
          f:kubernetes.io/arch: {}
          f:kubernetes.io/hostname: {}
          f:kubernetes.io/os: {}
          f:region: {}
      f:spec:
        f:providerID: {}
      f:status:
        f:addresses:
         .: {}
          k:{"type":"Hostname"}:
            .: {}
            f:address: {}
            f:type: {}
          k:{"type":"InternalIP"}:
            .: {}
            f:address: {}
            f:type: {}
        f:allocatable:
          .: {}
          f:cpu: {}
          f:ephemeral-storage: {}
          f:hugepages-2Mi: {}
          f:memory: {}
          f:pods: {}
        f:capacity:
          .: {}
          f:cpu: {}
          f:ephemeral-storage: {}
          f:hugepages-2Mi: {}
          f:memory: {}
          f:pods: {}
        f:conditions:
          .: {}
          k:{"type":"DiskPressure"}:
            .: {}
            f:lastHeartbeatTime: {}
            f:lastTransitionTime: {}
            f:message: {}
            f:reason: {}
            f:status: {}
            f:type: {}
          k:{"type":"MemoryPressure"}:
            .: {}
            f:lastHeartbeatTime: {}
            f:lastTransitionTime: {}
            f:message: {}
            f:reason: {}
            f:status: {}
            f:type: {}
          k:{"type":"PIDPressure"}:
            .: {}
            f:lastHeartbeatTime: {}
            f:lastTransitionTime: {}
            f:message: {}
            f:reason: {}
            f:status: {}
            f:type: {}
          k:{"type":"Ready"}:
            .: {}
            f:lastHeartbeatTime: {}
            f:lastTransitionTime: {}
            f:message: {}
            f:reason: {}
            f:status: {}
            f:type: {}
        f:daemonEndpoints:
          f:kubeletEndpoint:
         .: {}
          k:{"type":"Hostname"}:
            .: {}
            f:address: {}
            f:type: {}
          k:{"type":"InternalIP"}:
            .: {}
            f:address: {}
            f:type: {}
        f:allocatable:
          .: {}
          f:cpu: {}
          f:ephemeral-storage: {}
          f:hugepages-2Mi: {}
          f:memory: {}
          f:pods: {}
        f:capacity:
          .: {}
          f:cpu: {}
          f:ephemeral-storage: {}
          f:hugepages-2Mi: {}
          f:memory: {}
          f:pods: {}
        f:conditions:
          .: {}
          k:{"type":"DiskPressure"}:
            .: {}
            f:lastHeartbeatTime: {}
            f:lastTransitionTime: {}
            f:message: {}
            f:reason: {}
            f:status: {}
            f:type: {}
          k:{"type":"MemoryPressure"}:
            .: {}
            f:lastHeartbeatTime: {}
            f:lastTransitionTime: {}
            f:message: {}
            f:reason: {}
            f:status: {}
            f:type: {}
          k:{"type":"PIDPressure"}:
            .: {}
            f:lastHeartbeatTime: {}
            f:lastTransitionTime: {}
            f:message: {}
            f:reason: {}
            f:status: {}
            f:type: {}
          k:{"type":"Ready"}:
            .: {}
            f:lastHeartbeatTime: {}
            f:lastTransitionTime: {}
            f:message: {}
            f:reason: {}
            f:status: {}
            f:type: {}
        f:daemonEndpoints:
          f:kubeletEndpoint:
            f:Port: {}
        f:images: {}
        f:nodeInfo:
          f:architecture: {}
          f:bootID: {}
          f:containerRuntimeVersion: {}
          f:kernelVersion: {}
          f:kubeProxyVersion: {}
          f:kubeletVersion: {}
          f:machineID: {}
          f:operatingSystem: {}
          f:osImage: {}
          f:systemUUID: {}
    manager: kubelet
    operation: Update
    time: "2020-07-29T19:24:06Z"
  name: python-apps-3re6j
  resourceVersion: "2462715"
  selfLink: /api/v1/nodes/python-apps-3re6j
  uid: 6a72b4b6-38d7-4a27-bc32-220044c8004e
spec:
  podCIDR: 10.244.4.128/25
  podCIDRs:
  - 10.244.4.128/25
  providerID: digitalocean://201815603
  taints:
  - effect: NoSchedule
    key: node-type
    value: python-apps
  - effect: NoExecute
    key: node-type
    value: python-apps
status:
  addresses:
  - address: python-apps-3re6j
    type: Hostname
  - address: 10.136.206.175
    type: InternalIP
  - address: 104.248.120.49
    type: ExternalIP
  allocatable:
    cpu: "4"
    ephemeral-storage: "152161143761"
    hugepages-2Mi: "0"
    memory: 6694Mi
    pods: "110"
  capacity:
    cpu: "4"
    ephemeral-storage: 165105408Ki
    hugepages-2Mi: "0"
    memory: 8170048Ki
    pods: "110"
  conditions:
  - lastHeartbeatTime: "2020-07-28T21:00:04Z"
    lastTransitionTime: "2020-07-28T21:00:04Z"
    message: Cilium is running on this node
    reason: CiliumIsUp
    status: "False"
    type: NetworkUnavailable
  - lastHeartbeatTime: "2020-07-29T19:24:06Z"
    lastTransitionTime: "2020-07-28T20:59:46Z"
    message: kubelet has sufficient memory available
    reason: KubeletHasSufficientMemory
    status: "False"
    type: MemoryPressure
  - lastHeartbeatTime: "2020-07-29T19:24:06Z"
    lastTransitionTime: "2020-07-28T20:59:46Z"
    message: kubelet has no disk pressure
    reason: KubeletHasNoDiskPressure
    status: "False"
    type: DiskPressure
  - lastHeartbeatTime: "2020-07-29T19:24:06Z"
    lastTransitionTime: "2020-07-28T20:59:46Z"
    message: kubelet has sufficient PID available
    reason: KubeletHasSufficientPID
    status: "False"
    type: PIDPressure
  - lastHeartbeatTime: "2020-07-29T19:24:06Z"
    lastTransitionTime: "2020-07-28T20:59:56Z"
    message: kubelet is posting ready status
    reason: KubeletReady
    status: "True"
    type: Ready
  daemonEndpoints:
    kubeletEndpoint:
      Port: 10250
  nodeInfo:
    architecture: amd64
    bootID: 0df05998-4b67-490a-936e-961ec8156310
    containerRuntimeVersion: docker://18.9.2
    kernelVersion: 4.19.0-0.bpo.6-amd64
    kubeProxyVersion: v1.18.6
    kubeletVersion: v1.18.6
    machineID: fa1b1d9578c74b8583d7a73feb857047
    operatingSystem: linux
    osImage: Debian GNU/Linux 10 (buster)
    systemUUID: fa1b1d95-78c7-4b85-83d7-a73feb857047

node is a droplet on DO. I removed .images[] block

Looks like this PR fixes it: #13

kvaps commented

Okay, accepted.

I was thinking that all taints are ignored if .spec.nodeName is specified for the pod, but I was wrong NoExecute is still respected.

Fixed in 1c089c3, thank you guys! 👍