Where are the backups actually stored?

[maxisme]

Hello,

I followed the install process of volimesnapshot from kubernetes - https://github.com/kubernetes-csi/external-snapshotter

I then setup your end and I now have what appear to be multiple backups:

➜ kubectl get VolumeSnapshot       
NAME                     AGE
test-volume-1614127971   13h
test-volume-1614171184   103m
test-volume-1614174784   43m
test-volume-1614175384   33m
test-volume-1614175984   23m
test-volume-1614176584   13m
test-volume-1614177184   3m55s

I am however a bit confused as to where they are actually backed up? I was expecting something to be returned from kubectl get VolumeSnapshotContent ? (My end goal is to create backups of the backups 🙂)

[rbren]

This depends on your cluster and your CSI provider. For me, the backups are stored in DigitalOcean as Snapshots.

You should definitely see results for kubectl get volumesnapshotcontent. Are the volumesnapshots showing status=Ready? They might be stuck.

[maxisme]

I actually just upgraded and now I have this sort of format:

➜ kubectl get VolumeSnapshot
NAME                     READYTOUSE   SOURCEPVC   SOURCESNAPSHOTCONTENT   RESTORESIZE   SNAPSHOTCLASS   SNAPSHOTCONTENT   CREATIONTIME   AGE
test-volume-1614178671                kibana                                                                                             89m
test-volume-1614181671                kibana                                                                                             39m
test-volume-1614182271                kibana                                                                                             29m
test-volume-1614182871                kibana                                                                                             19m
test-volume-1614183471                kibana                                                                                             9m40s

Where can I find status=Ready sorry? New to k8s.

[maxisme]

This is what I am running to install:

git clone https://github.com/kubernetes-csi/external-snapshotter
kubectl create -f external-snapshotter/client/config/crd
kubectl apply -f external-snapshotter/deploy/kubernetes/snapshot-controller
kubectl apply -f external-snapshotter/deploy/kubernetes/csi-snapshotter
rm -rf external-snapshotter

helm repo add fairwinds-stable https://charts.fairwinds.com/stable
helm install gemini fairwinds-stable/gemini
kubectl apply -f snapshot-group.yml

snapshot-group.yml:

apiVersion: gemini.fairwinds.com/v1beta1
kind: SnapshotGroup
metadata:
  name: test-volume
spec:
  persistentVolumeClaim:
    claimName: kibana
  schedule:
    - every: 1 minutes
      keep: 3

[maxisme]

I seem to be getting the error: checkAndUpdateSnapshotClass failed to setDefaultClass snapshotting non-CSI volumes is not supported, snapshot:default/test-volume-1614186855

[rbren]

🤔 looks like your PVC isn't a CSI-friendly volume. What does

kubectl describe pvc kibana

show? Also interested in

kubectl describe pv kibana

(assuming that's the name of the PV)

[lalet]

I seem to be getting the error: checkAndUpdateSnapshotClass failed to setDefaultClass snapshotting non-CSI volumes is not supported, snapshot:default/test-volume-1614186855

How did you figure out this log ? I am also trying to create a backup of a volume on an existing pvc, but I can not see any action performed on it.

[rbren]

You can check the logs of your CSI driver, as well as for gemini itself. There might be some clues in there.

[lalet]

You can check the logs of your CSI driver, as well as for gemini itself. There might be some clues in there.

E0331 19:38:13.888884 1 controller.go:164] jenkins/test-jenkins: failed to perform backup - jenkins/test-jenkins: could not find existing PVC pvc-2276b5e3-6003-4cd3-902c-194642e272cc
E0331 19:38:13.888934 1 controller.go:146] jenkins/test-jenkins: error syncing controller.workItem{name:"test-jenkins", namespace:"jenkins", snapshotGroup:(*v1.SnapshotGroup)(0xc0001a8380), task:0}: jenkins/test-jenkins: could not find existing PVC pvc-2276b5e3-6003-4cd3-902c-194642e272cc, requeuing
I0331 19:38:13.888950 1 groups.go:39] default/mypv-backup: reconciling
E0331 19:38:13.892019 1 controller.go:164] default/mypv-backup: failed to perform backup - default/mypv-backup: could not find existing PVC mypv
E0331 19:38:13.892084 1 controller.go:146] default/mypv-backup: error syncing controller.workItem{name:"mypv-backup", namespace:"default", snapshotGroup:(*v1.SnapshotGroup)(0xc000010fc0), task:0}: default/mypv-backup: could not find existing PVC mypv, requeuing

It is not able to find an existing pvc for some reason, but for the pvc I created using gemini it is working fine though.

[rbren]

what's the output of

kubectl get pv

and

kubectl get pvc -n default

?

[lalet]

I was trying to create backup for jenkins/jenkins-home3-claim (pvc-2276b5e3-6003-4cd3-902c-194642e272cc)

[rbren]

🤔 not sure why it can't find them.

Do the manual and standard storage classes have volumesnapshot support?

[lalet]

I was able to create backup of a pvc pvc-demo (standard storage class), the one I am trying is of a persistent volume, gemini supports persistent volume backup ?

[lalet]

🤔 not sure why it can't find them.

Do the manual and standard storage classes have volumesnapshot support?

Yes https://kubernetes.io/docs/concepts/storage/volume-snapshots/, I was able to use this to create snapshot of pvc-2276b5e3-6003-4cd3-902c-194642e272cc

[lalet]

But nonetheless, this is an awesome tool, thank you!

[rbren]

I was able to create backup of a pvc pvc-demo (standard storage class), the one I am trying is of a persistent volume, gemini supports persistent volume backup ?

🤔 IIRC you can only take a volumesnapshot of a PVC, not a PV. I could be wrong though.

[lalet]

My bad, the pvc was in the jenkins namespace, and there was a similar named persistent volume that was in the default namespace and hence all this confusion. I am able to create snapshotvolumes now using gemini! :)

[rbren]

Glad you figured it out!