/cloud-director-named-disk-csi-driver

Container Storage Interface (CSI) driver for VMware Cloud Director

Primary LanguageGoOtherNOASSERTION

Container Storage Interface (CSI) driver for VMware Cloud Director Named Independent Disks

This repository contains the source code and build methods to build a Kubernetes CSI driver that helps provision VMware Cloud Director Named Independent Disks as a storage solution for Kubernetes Applications. This uses VMware Cloud Director API for functionality and hence needs an appropriate VMware Cloud Director Installation. This CSI driver will help enable common scenarios with persistent volumes and stateful-sets using VMware Cloud Director Shareable Named Disks.

The version of the VMware Cloud Director API and Installation that are compatible for a given CSI container image are provided in the following compatibility matrix:

CSI Version CSE Version VMware Cloud Director API VMware Cloud Director Installation Notes Kubernetes Versions docs
main 4.1.z 36.0+ 10.3.3.4+
  • Bump gopkg.in/yaml.v3 version (#221)
  • Changes to testing framework (multiple PR's)
  • 1.25
  • 1.24
  • 1.23
  • 1.22
  • 1.21
 CSI main docs
1.4.1 4.1.0 36.0+ 10.3.3.4+
  • Bump gopkg.in/yaml.v3 version (#221)
  • Changes to testing framework (multiple PR's)
  • 1.25
  • 1.24
  • 1.23
  • 1.22
  • 1.21
 CSI 1.4.z docs
1.4.0 4.1.0 36.0+ 10.3.3.4+
  • Support for packaging CSI CRS in a container for CSE airgap workflow
  • Testing Framework added
  • Run CSI only on Control-plane Nodes
  • Change CSI controller from stateful-set to deployment
  • Support newer capvcdCluster RDE version
  • Fix issues in XFS mount (support XFS)
  • Set description of named disks as ClusterID
  • upgrade golang version to 1.19
  • optimize image size of CSI container image
  • 1.25
  • 1.24
  • 1.23
  • 1.22
  • 1.21
 CSI 1.4.z docs
1.3.2 4.0.z 36.0+ 10.3.1+
(10.3.1 needs hot-patch to prevent VCD cell crashes in multi-cell environments)
  • 1.22
  • 1.21
  • 1.20
  • 1.19
 CSI 1.3.z docs
1.3.1 4.0.0 36.0+ 10.3.1+
(10.3.1 needs hot-patch to prevent VCD cell crashes in multi-cell environments)
  • Fixed issue where CSI failed to mount persistent volume to node if SCSI Buses inside node are not rescanned
  • 1.22
  • 1.21
  • 1.20
  • 1.19
 CSI 1.3.z docs
1.3.0 4.0.0 36.0+ 10.3.1+
(10.3.1 needs hot-patch to prevent VCD cell crashes in multi-cell environments)
  • Support for fsGroup
  • Support for volume metrics
  • Added secret-based way to get cluster-id for CRS
  • 1.22
  • 1.21
  • 1.20
  • 1.19
 CSI 1.3.z docs
1.2.1 3.1.x 36.0+ 10.3.1+
(10.3.1 needs hot-patch to prevent VCD cell crashes in multi-cell environments)
  • 1.22
  • 1.21
  • 1.20
  • 1.19
 CSI 1.2.x docs
1.2.0 3.1.x 36.0+ 10.3.1+
(10.3.1 needs hot-patch to prevent VCD cell crashes in multi-cell environments)
  • Add support for Kubernetes 1.22
  • Small VCD url parsing fixes
  • 1.22
  • 1.21
  • 1.20
  • 1.19
 CSI 1.2.x docs
1.1.1 3.1.x 36.0+ 10.3.1+
(10.3.1 needs hot-patch to prevent VCD cell crashes in multi-cell environments)
  • Fixed refresh-token based authentication issue observed when VCD cells are fronted by a load balancer (Fixes #26).
  • 1.21
  • 1.20
  • 1.19
 CSI 1.1.x docs
1.1.0 3.1.x 36.0+ 10.3.1+
(10.3.1 needs hot-patch to prevent VCD cell crashes in multi-cell environments)
  • Remove legacy Kubernetes dependencies.
  • Support for CAPVCD RDEs.
  • 1.21
  • 1.20
  • 1.19
 CSI 1.1.x docs
1.0.0 3.1.x 36.0+ 10.3.1+
(10.3.1 needs hot-patch to prevent VCD cell crashes in multi-cell environments)		 First cut with support for Named Independent Disks
  • 1.21
  • 1.20
  • 1.19
 CSI 1.0.0 docs

This extension is intended to be installed into a Kubernetes cluster installed with VMware Cloud Director as a Cloud Provider, by a user that has the rights as described in the sections below.

cloud-director-named-disk-csi-driver is distributed as a container image hosted at Distribution Harbor as projects.registry.vmware.com/vmware-cloud-director/cloud-director-named-disk-csi-driver:<CSI version>

This driver is in a GA state and will be supported in production.

Note: This driver is not impacted by the Apache Log4j open source component vulnerability.

CSI Feature matrix

Feature Support Scope
Storage Type
    Independent Shareable Named Disks of VCD
Provisioning
  • Static Provisioning
  • Dynamic Provisioning
Access Modes
  • ReadOnlyMany
  • ReadWriteOnce
Volume
    Block
VolumeMode
  • FileSystem
Volume Expansion Support
  • OFFLINE
  • ONLINE
Topology
  • Static Provisioning: reuses VCD topology capabilities
  • Dynamic Provisioning: places disk in the OVDC of the ClusterAdminUser based on the StorageProfile specified.

Terminology

  1. VCD: VMware Cloud Director
  2. ClusterAdminRole: This is the role that has enough rights to create and administer a Kubernetes Cluster in VCD. This role can be created by cloning the vApp Author Role and then adding the following rights (details on adding the rights below can be found in the CSE docs):
    1. Full Control: VMWARE:CAPVCDCLUSTER
    2. Edit: VMWARE:CAPVCDCLUSTER
    3. View: VMWARE:CAPVCDCLUSTER
  3. ClusterAdminUser: For CSI functionality, there needs to be a set of additional rights added to the ClusterAdminRole as described in the "Additional Rights for CSI" section below. The Kubernetes Cluster needs to be created by a user belonging to this enhanced ClusterAdminRole. For convenience, let us term this user as the ClusterAdminUser.

VMware Cloud Director Configuration

In this section, we assume that the Kubernetes cluster is created using the Container Service Extension 4.0. However, that is not a mandatory requirement.

Additional Rights for CSI

The ClusterAdminUser should have view access to the vApp containing the Kubernetes cluster. Since the ClusterAdminUser itself creates the cluster, it will have this access by default. This ClusterAdminUser needs to be created from a ClusterAdminRole with the following additional rights:

  1. Access Control =>
    1. User => Manage user's own API TOKEN
  2. Organization VDC => Create a Shared Disk

Troubleshooting

Log VCD requests and responses

Execute the following command to log HTTP requests to VCD and HTTP responses from VCD -

kubectl set env -n kube-system StatefulSet/csi-vcd-controllerplugin -c vcd-csi-plugin GOVCD_LOG_ON_SCREEN=true -oyaml
kubectl set env -n kube-system DaemonSet/csi-vcd-nodeplugin -c vcd-csi-plugin GOVCD_LOG_ON_SCREEN=true -oyaml

Once the above command is executed, CSI containers will start logging the HTTP requests and HTTP responses made via go-vcloud-director SDK. The container logs can be obtained using the command kubectl logs -n kube-system <CSI pod name>

To stop logging the HTTP requests and responses from VCD, the following command can be executed -

kubectl set env -n kube-system Deployment/csi-vcd-controllerplugin -c vcd-csi-plugin GOVCD_LOG_ON_SCREEN-
kubectl set env -n kube-system DaemonSet/csi-vcd-nodeplugin -c vcd-csi-plugin GOVCD_LOG_ON_SCREEN-

NOTE: Please make sure to collect the logs before and after enabling the wire log. The above commands update the CSI controller Deployment and CSI node-plugin DaemonSet, which creates a new CSI pods. The logs present in the old pods will be lost.

Upgrade CSI

To perform an upgrade of the Container Storage Interface (CSI) from versions v1.2.0, v1.2.1, v1.3.0, v1.3.1, and v1.3.2, it is recommended to follow the following steps:

  1. Remove the current StatefulSet:
kubectl delete statefulset -n kube-system csi-vcd-controllerplugin
  1. Apply the CSI 1.4 Controller CRS:
kubectl apply -f https://github.com/vmware/cloud-director-named-disk-csi-driver/blob/1.4.z/manifests/csi-controller-crs.yaml

NOTE:

  1. These steps ensure a successful upgrade of CSI to the latest version (v1.4.0) and guarantee that the new CSI Deployment is properly installed within the Kubernetes environment.
  2. it is recommended not to manually delete any Persistent Volumes (PVs) or Persistent Volume Claims (PVCs) associated with a StatefulSet.

Contributing

Please see CONTRIBUTING.md for instructions on how to contribute.

License

Apache-2.0