/csi-baremetal-operator

Primary LanguageGoApache License 2.0Apache-2.0

PR validation codecov Go Report Card Kubernetes

Bare-metal CSI Operator

Kubernetes Operator to deploy and manage lifecycle of Bare-Metal CSI Driver

Supported environments

  • OpenShift: 4.6
  • RKE2r1: 1.25-1.28
  • Node OS:
    • Red Hat Enterprise Linux 7.7 / 8.x
    • Red Hat Enterprise Linux CoreOS 4.6
    • CentOS Linux 8
  • Helm: 3.0

Installation process

  • Prerequisites

    • lvm2 packet installed on the Kubernetes nodes
    • helm v3+
    • kubectl v1.16+
  • Add helm repository

    helm repo add csi https://dell.github.io/csi-baremetal-operator
    helm repo update
    helm search repo csi -l
  • Setup environment variables

    export REGISTRY=docker.io/objectscale
    export DOCKER_REGISTRY_SECRET=dockerhub-pull-secret
  • Create docker registry secret

    kubectl create secret docker-registry $DOCKER_REGISTRY_SECRET --docker-username=<USER NAME> \
    --docker-password=<PASSWORD> --docker-email=<EMAIL>
  • Install CSI Operator

    helm install csi-baremetal-operator csi/csi-baremetal-operator --set global.registry=$REGISTRY \
    --set global.registrySecret=$DOCKER_REGISTRY_SECRET
  • Install CSI

    • Vanilla Kubernetes

      helm install csi-baremetal csi/csi-baremetal-deployment --set scheduler.patcher.enable=true \
      --set driver.drivemgr.type=halmgr --set global.registry=$REGISTRY --set global.registrySecret=$DOCKER_REGISTRY_SECRET
      
    • RKE2

      helm install csi-baremetal csi/csi-baremetal-deployment --set scheduler.patcher.enable=true \
      --set driver.drivemgr.type=halmgr --set platform=rke --set global.registry=$REGISTRY \
      --set global.registrySecret=$DOCKER_REGISTRY_SECRET
      
    • OpenShift

      helm install csi-baremetal csi/csi-baremetal-deployment --set scheduler.patcher.enable=true \
      --set driver.drivemgr.type=halmgr --set platform=openshift --set global.registry=$REGISTRY \
      --set global.registrySecret=$DOCKER_REGISTRY_SECRET
      

      Note: In case of CSI deployment at Openshift platform in non default namespace, CSI ServiceAccounts
      csi-node-sa and csi-baremetal-extender-sa will require Privileged SCC. In case there are no such permissions
      CSI Operator will not deploy node and scheduler extender daemonsets. You can see the following events in such case:

      65m         Warning   SecurityContextConstraintsVerificationFailed       deployment/csi-baremetal                              ServiceAccount csi-node-sa has insufficient securityContextConstraints, should have privileged
      65m         Warning   SecurityContextConstraintsVerificationFailed       deployment/csi-baremetal                              ServiceAccount csi-baremetal-extender-sa has insufficient securityContextConstraints, should have privileged
      

      To make them deployable - create the following role and rolebinding in deployed namespace:

      apiVersion: rbac.authorization.k8s.io/v1
      kind: Role
      metadata:
        name: $ROLENAME
        namespace: $NAMESPACE
      rules:
      - apiGroups:
        - security.openshift.io
        resourceNames:
        - privileged
        resources:
        - securitycontextconstraints
        verbs:
        - use
      apiVersion: rbac.authorization.k8s.io/v1
      kind: RoleBinding
      metadata:
        name: role-binding
        namespace: $NAMESPACE
      roleRef:
        apiGroup: rbac.authorization.k8s.io
        kind: Role
        name: $ROLENAME
      subjects:
      - kind: ServiceAccount
        name: csi-baremetal-extender-sa
        namespace: $NAMESPACE
      - kind: ServiceAccount
        name: csi-node-sa
        namespace: $NAMESPACE

      After created the below roles and rolebindings CSI Operator will automatically catch the changes and deploy node and scheduler extender daemonsets.

    • Kind (for testing purposes only)

      helm install csi-baremetal csi/csi-baremetal-deployment --set scheduler.patcher.enable=true \
      --set driver.drivemgr.type=loopbackmgr --set driver.drivemgr.deployConfig=true --set global.registry=$REGISTRY \
      --set global.registrySecret=$DOCKER_REGISTRY_SECRET
      
    • K3S

      helm install csi-baremetal csi/csi-baremetal-deployment --set driver.drivemgr.type=halmgr \
      --set global.registry=$REGISTRY --set global.registrySecret=$DOCKER_REGISTRY_SECRET
      

      Note: Currently supports only manual patching - refer documentation

    • Not supported platform or system with third party Kubernetes scheduler extender - refer documentation for manual patching of Kubernetes scheduler configuration

      helm install csi-baremetal csi/csi-baremetal-deployment --set driver.drivemgr.type=halmgr \
      --set global.registry=$REGISTRY --set global.registrySecret=$DOCKER_REGISTRY_SECRET
      

Feature Supporting

CIS hardening

For supporting CIS Hardening we need to add the following settings while installing CSI and CSI Baremetal Operator:

  • While installing CSI Baremetal Operator set SecurityContext:
    --set securityContext.enable=true --set securityContext.runAsNonRoot=true --set securityContext.runAsUser=1000
    
  • While installing CSI:
    • Set the following SecurityContexts at deployments.csi-baremetal.dell.com for Controller, NodeController components:
      --set driver.controller.securityContext.enable=true --set driver.controller.securityContext.runAsNonRoot=true --set driver.controller.securityContext.runAsUser=1000 \
      --set driver.securityContext.securityContext.enable=true --set driver.securityContext.securityContext.runAsNonRoot=true --set driver.securityContext.securityContext.runAsUser=1000
      
    • Create (if not existed) privileged PodSecurityPolicy:
      apiVersion: policy/v1beta1
      kind: PodSecurityPolicy
      metadata:
        name: privileged
        annotations:
          seccomp.security.alpha.kubernetes.io/allowedProfileNames: '*'
      spec:
        privileged: true
        allowPrivilegeEscalation: true
        allowedCapabilities:
        - '*'
        volumes:
        - '*'
        hostNetwork: true
        hostPorts:
        - min: 0
          max: 65535
        hostIPC: true
        hostPID: true
        runAsUser:
          rule: 'RunAsAny'
        seLinux:
          rule: 'RunAsAny'
        supplementalGroups:
          rule: 'RunAsAny'
        fsGroup:
          rule: 'RunAsAny'
      
    • Create Role for this policy usage:
      apiVersion: rbac.authorization.k8s.io/v1
      kind: Role
      metadata:
        name: manager-role
        namespace: test-namespace
      rules:
      - apiGroups:
        - policy
          resourceNames:
        - privileged
          resources:
        - podsecuritypolicies
          verbs:
        - use
      
    • Create RoleBinding for this policy usage:
      apiVersion: rbac.authorization.k8s.io/v1
      kind: RoleBinding
      metadata:
        name: manager-rolebinding
        namespace: test-namespace
      roleRef:
        apiGroup: rbac.authorization.k8s.io
        kind: Role
        name: manager-role
      subjects:
      - kind: ServiceAccount
        name: csi-node-sa
        namespace: test-namespace
      - kind: ServiceAccount
        name: csi-baremetal-extender-sa
        namespace: test-namespace
      
    • Set the following SecurityContexts at deployments.csi-baremetal.dell.com for SchedulerExtender, SchedulerExtenderPatcher components:
      --set scheduler.securityContext.enable=true --set scheduler.securityContext.privileged=true
      
    • Enable podSecurityPolicy at deployments.csi-baremetal.dell.com for Controller, Node, SchedulerExtender, SchedulerExtenderPatcher components:
      --set driver.controller.securityContext.enable=true --set driver.controller.securityContext.runAsNonRoot=true --set driver.controller.securityContext.runAsUser=1000 \
      --set driver.node.podSecurityPolicy.enable=true --set driver.node.podSecurityPolicy.resourceName=privileged \
      --set scheduler.podSecurityPolicy.enable=true --set scheduler.podSecurityPolicy.resourceName=privileged
      

Usage

  • Storage classes

    • Use storage class without lvg postfix if you need to provision PV bypassing LVM. Size of the resulting PV will be equal to the size of underlying physical drive.

    • Use storage class with lvg postfix if you need to provision PVC based on the logical volume. Size of the resulting PV will be equal to the size of PVC.

  • To obtain information about:

    • Node IDs assigned by CSI - kubectl get nodes.csi-baremetal.dell.com

    • Local Drives discovered by CSI - kubectl get drives.csi-baremetal.dell.com

    • Capacity available for allocation - kubectl get availablecapacities.csi-baremetal.dell.com

    • Provisioned logical volume groups - kubectl get logicalvolumegroups.csi-baremetal.dell.com

    • Provisioned volumes - kubectl get volumes.csi-baremetal.dell.com

    • Storage groups - kubectl get sgs

Upgrade process

To upgrade please reference Installation process section but replace helm install by helm upgrade command.

See helm upgrade for command documentation.

Uninstallation process

  • Delete all Bare-metal-CSI-managed PVCs
    pvcs=($(kubectl get pvc -o wide -A | grep csi-baremetal-sc | awk {'print $1" "$2'}))
    for (( i=0; i<${#pvcs[@]} ; i+=2 ))
    do
      kubectl delete pvc ${pvcs[i+1]} -n ${pvcs[i]}
    done
    
  • Ensure there is no Bare-metal-CSI-managed PVC left
    kubectl get pvc -o wide -A | grep csi-baremetal-sc
    
  • Delete custom resources
    kubectl delete volumes --all -A
    kubectl delete lvgs --all
    kubectl delete sgs --all
    kubectl delete csibmnodes --all
    
  • Delete helm releases
    helm delete csi-baremetal
    helm delete csi-baremetal-operator
    
  • Delete custom resource definitions
    kubectl delete crd deployments.csi-baremetal.dell.com availablecapacities.csi-baremetal.dell.com \
    availablecapacityreservations.csi-baremetal.dell.com logicalvolumegroups.csi-baremetal.dell.com \
    volumes.csi-baremetal.dell.com drives.csi-baremetal.dell.com nodes.csi-baremetal.dell.com \
    storagegroups.csi-baremetal.dell.com