openebs-archive/openebs-docs

Update the steps for cstor volume resize.

kmova opened this issue · 2 comments

kmova commented

From OpenEBS 1.2 onwards how to perform resize of cStor Volume

Step1: Find the cstorvolume of pvc which needs to perform PVC resize(kubect get cstorvolume -n <openebs_namespace> -l openebs.io/persistent-volume=<pv_name>).

Step2: Update the cstorvolume spec.Capacity with the new size(Wait till the status.capacity of cstorvolume is updated to the latest size).

Step3: https://github.com/openebs/openebs-docs/blob/day_2_ops/docs/cstor_volume_resize.md#rescan-iscsi-on-node-where-application-is-running (As mentioned in the above link following manual steps are required to rescan of iscsi session, resizing filesystem, verify whether size got reflected inside the application pod and at last updating PV capacity)

mittachaitu commented

Below are the detailed steps to perform cStor volume resize and If OpenEBS version >= 1.2 then follow the below steps. Click here to perform resize if OpenEBS version is < OpenEBS1.2

Step1: Find the cstorvolume of corresponding PV

kubectl get cstorvolume -n openebs -l openebs.io/persistent-volume=pvc-15f6b83a-4ce7-471c-9fdd-64d092ddf105(pv name)
NAME                                       STATUS    AGE    CAPACITY
pvc-15f6b83a-4ce7-471c-9fdd-64d092ddf105   Healthy   105m   5Gi

Step2: Edit spec.capacity in the cStorVolume

kubectl edit cstorvolume pvc-15f6b83a-4ce7-471c-9fdd-64d092ddf105 -n openebs

Step3: Wait still the status.capacity was updated to latest capacity (or) events on cstorvolume will say resize was done

From Status.Capacity
 kubectl get cstorvolume pvc-15f6b83a-4ce7-471c-9fdd-64d092ddf105 -n openebs
NAME                                       STATUS    AGE    CAPACITY
pvc-15f6b83a-4ce7-471c-9fdd-64d092ddf105   Healthy   109m   10Gi

Events

Events:
  Type    Reason   Age                From                                                                        Message
  ----    ------   ----               ----                                                                        -------
  Normal  Updated  2s (x3 over 111m)  pvc-15f6b83a-4ce7-471c-9fdd-64d092ddf105-target-777c9c8c7cgbd94, 127.0.0.1  Updated resize conditions
  Normal  Updated  1s                 pvc-15f6b83a-4ce7-471c-9fdd-64d092ddf105-target-777c9c8c7cgbd94, 127.0.0.1  successfully resized volume from 5Gi to 10Gi

Resizing cStorvolume was done with above steps. To reflect this size in application perform below steps

Rescan iscsi on node where application is running

Find correct node

$ kubectl -n <APP_NAMESPACE> get pods -o wide

NAME                                   READY   STATUS    RESTARTS   AGE   IP           NODE                      NOMINATED NODE   READINESS GATES
percona-test-percona-db78ff4d8-qztph   1/1     Running   0          55m   10.244.3.7   k8s-openebs-demo-node-0   <none>           <none>

Connect to node (ssh, gcloud compute ssh, etc) and run the following commands

Follow your standard procedures for connecting to your node. See instructions from your cloud provider regarding SSH for more information.

Rescan iscsi
$ sudo iscsiadm -m node -R

Rescanning session [sid: 2, target: iqn.2016-09.com.openebs.cstor:pvc-15f6b83a-4ce7-471c-9fdd-64d092ddf105, portal: 10.103.75.24,3260]
Verify size using lsblk and get device name
$ lsblk

NAME    MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
loop0     7:0    0 88.4M  1 loop /snap/core/6964
loop1     7:1    0 58.9M  1 loop /snap/google-cloud-sdk/84
loop2     7:2    0 59.1M  1 loop /snap/google-cloud-sdk/85
sda       8:0    0   20G  0 disk 
├─sda1    8:1    0 19.9G  0 part /
├─sda14   8:14   0    4M  0 part 
└─sda15   8:15   0  106M  0 part /boot/efi
sdb       8:16   0   50G  0 disk 
├─sdb1    8:17   0   50G  0 part 
└─sdb9    8:25   0    8M  0 part 
sdc       8:32   0   50G  0 disk 
├─sdc1    8:33   0   50G  0 part 
└─sdc9    8:41   0    8M  0 part 
sdd       8:48   0   10G  0 disk /var/lib/kubelet/pods/ce5888b6-87b8-11e9-b062-42010a800014/volumes/kubernetes.io~iscsi/pvc-15f6b83a-4ce7-471c-9fdd-64d092ddf105
Resize the filesystem

The device name was listed in the last command. That will be used here for the resize.

$ sudo resize2fs /dev/<DEVICE_NAME>

resize2fs 1.44.1 (24-Mar-2018)
Filesystem at /dev/sdd is mounted on /var/lib/kubelet/plugins/kubernetes.io/iscsi/iface-default/10.103.75.24:3260-iqn.2016-09.com.openebs.cstor:pvc-15f6b83a-4ce7-471c-9fdd-64d092ddf105-lun-0; on-line resizing required
old_desc_blocks = 1, new_desc_blocks = 3
The filesystem on /dev/sdd is now 2621440 (4k) blocks long.

Verify size in application pod

$ kubectl -n <APP_NAMESPACE> exec -it <APP_POD> -- df -h

Filesystem      Size  Used Avail Use% Mounted on
overlay          20G  3.6G   16G  19% /
tmpfs            64M     0   64M   0% /dev
tmpfs           1.8G     0  1.8G   0% /sys/fs/cgroup
/dev/sda1        20G  3.6G   16G  19% /etc/hosts
shm              64M     0   64M   0% /dev/shm
/dev/sdd         10G  258M   20G   2% /var/lib/mysql
tmpfs           1.8G   12K  1.8G   1% /run/secrets/kubernetes.io/serviceaccount
tmpfs           1.8G     0  1.8G   0% /proc/acpi
tmpfs           1.8G     0  1.8G   0% /proc/scsi
tmpfs           1.8G     0  1.8G   0% /sys/firmware

Edit PV

Get PV

$ kubectl get pv

NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                          STORAGECLASS         REASON   AGE
pvc-15f6b83a-4ce7-471c-9fdd-64d092ddf105   5Gi       RWO            Delete           Bound    default/percona-test-percona   openebs-cstor-test            46m

Patch PV

Now edit spec.capacity.storage in the PV

NOTE: PVC size will remain the same as the old size(but that will not be a problem for application) but the application can consume the expanded size.

mittachaitu commented

Document has been updated via #755