libopenstorage/stork

Malformed snapshot, failed to get volume plugin

cehoffman opened this issue · 2 comments

cehoffman commented

Is this a BUG REPORT or FEATURE REQUEST?: BUG REPORT

What happened:

Creating snapshot from pvc failed to provision with malformed snapshot.

What you expected to happen:

A snapshot was made so a PVC could be bound.

How to reproduce it (as minimally and precisely as possible):

  1. Create VolumeSnapshot

Anything else we need to know?:

Environment:

  • Kubernetes version (use kubectl version): 1.8.7_coreos.0
  • Cloud provider or hardware configuration: Azure
  • OS (e.g. from /etc/os-release): 1632.3.0
  • Kernel (e.g. uname -a):
  • Install tools:
  • Others: Stork 1.0.2

Below is the log from stork, the VolumeSnapshot in question, the PVC it references, and the PV used by that PVC.

I0309 19:35:56.212010       1 snapshotter.go:448] findSnapshot: snapshot review-bump-10-5-ob7wul-redis
E0309 19:35:56.212032       1 snapshotter.go:455] Failed to get volume plugin. Snapshot review-bump-10-5-ob7wul-redis is malformed
I0309 19:35:56.212042       1 snapshotter.go:439] syncSnapshot: Creating snapshot gitlab/review-bump-10-5-ob7wul-redis ...
I0309 19:35:56.212047       1 snapshotter.go:481] createSnapshot: Creating snapshot gitlab/review-bump-10-5-ob7wul-redis through the plugin ...
I0309 19:35:56.223428       1 controller.go:1080] scheduleOperation[lock-provision-gitlab/review-bump-10-5-ob7wul-redis[f34ae5d8-23d0-11e8-b26a-000d3a031a10]]
I0309 19:35:56.237821       1 snapshotter.go:487] createSnapshot: Creating metadata for snapshot gitlab/review-bump-10-5-ob7wul-redis.
I0309 19:35:56.237887       1 snapshotter.go:690] In updateVolumeSnapshotMetadata
I0309 19:35:56.249019       1 snapshotter.go:717] updateVolumeSnapshotMetadata: Metadata UID: f3463bc3-23d0-11e8-b26a-000d3a031a10 Metadata Name: review-bump-10-5-ob7wul-redis Metadata Namespace: gitlab Setting tags in Metadata Labels: map[string]string{"SnapshotMetadata-Timestamp":"1520624156249013957", "SnapshotMetadata-PVName":"pvc-e0fc50db-8d3d-11e7-b368-000d3a035c62"}.
I0309 19:35:56.255096       1 leaderelection.go:156] attempting to acquire leader lease...
I0309 19:35:56.255768       1 snapshot-controller.go:203] [CONTROLLER] OnUpdate oldObj: v1.VolumeSnapshotSpec{PersistentVolumeClaimName:"millions-redis", SnapshotDataName:""}
I0309 19:35:56.255886       1 snapshot-controller.go:204] [CONTROLLER] OnUpdate newObj: v1.VolumeSnapshotSpec{PersistentVolumeClaimName:"millions-redis", SnapshotDataName:""}
I0309 19:35:56.256953       1 snapshotter.go:737] updateVolumeSnapshotMetadata: returning cloudTags [map[string]string{"kubernetes.io/created-for/snapshot/name":"review-bump-10-5-ob7wul-redis", "kubernetes.io/created-for/snapshot/uid":"f3463bc3-23d0-11e8-b26a-000d3a031a10", "kubernetes.io/created-for/snapshot/timestamp":"1520624156249013957", "kubernetes.io/created-for/snapshot/namespace":"gitlab"}]
time="2018-03-09T19:35:56Z" level=debug msg="SnapshotCreate for pv: &PersistentVolume{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:pvc-e0fc50db-8d3d-11e7-b368-000d3a035c62,GenerateName:,Namespace:,SelfLink:/api/v1/persistentvolumes/pvc-e0fc50db-8d3d-11e7-b368-000d3a035c62,UID:e32af46f-8d3d-11e7-b368-000d3a035c62,ResourceVersion:168985056,Generation:0,CreationTimestamp:2017-08-30 04:44:20 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{},Annotations:map[string]string{kubernetes.io/createdby: portworx-volume-dynamic-provisioner,pv.kubernetes.io/bound-by-controller: yes,pv.kubernetes.io/provisioned-by: kubernetes.io/portworx-volume,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,},Spec:PersistentVolumeSpec{Capacity:ResourceList{storage: {{10737418240 0} {<nil>} 10Gi BinarySI},},PersistentVolumeSource:PersistentVolumeSource{GCEPersistentDisk:nil,AWSElasticBlockStore:nil,HostPath:nil,Glusterfs:nil,NFS:nil,RBD:nil,ISCSI:nil,Cinder:nil,CephFS:nil,FC:nil,Flocker:nil,FlexVolume:nil,AzureFile:nil,VsphereVolume:nil,Quobyte:nil,AzureDisk:nil,PhotonPersistentDisk:nil,PortworxVolume:&PortworxVolumeSource{VolumeID:805925391547708106,FSType:,ReadOnly:false,},ScaleIO:nil,Local:nil,StorageOS:nil,},AccessModes:[ReadWriteOnce],ClaimRef:&ObjectReference{Kind:PersistentVolumeClaim,Namespace:gitlab,Name:millions-redis,UID:e0fc50db-8d3d-11e7-b368-000d3a035c62,APIVersion:v1,ResourceVersion:168985004,FieldPath:,},PersistentVolumeReclaimPolicy:Delete,StorageClassName:ext4-2x-low,MountOptions:[],},Status:PersistentVolumeStatus{Phase:Bound,Message:,Reason:,},} \n tags: &map[kubernetes.io/created-for/snapshot/namespace:gitlab kubernetes.io/created-for/snapshot/name:review-bump-10-5-ob7wul-redis kubernetes.io/created-for/snapshot/uid:f3463bc3-23d0-11e8-b26a-000d3a031a10 kubernetes.io/created-for/snapshot/timestamp:1520624156249013957]"
W0309 19:35:56.433679       1 snapshotter.go:268] failed to snapshot &v1.PersistentVolumeSpec{Capacity:v1.ResourceList{"storage":resource.Quantity{i:resource.int64Amount{value:10737418240, scale:0}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"10Gi", Format:"BinarySI"}}, PersistentVolumeSource:v1.PersistentVolumeSource{GCEPersistentDisk:(*v1.GCEPersistentDiskVolumeSource)(nil), AWSElasticBlockStore:(*v1.AWSElasticBlockStoreVolumeSource)(nil), HostPath:(*v1.HostPathVolumeSource)(nil), Glusterfs:(*v1.GlusterfsVolumeSource)(nil), NFS:(*v1.NFSVolumeSource)(nil), RBD:(*v1.RBDVolumeSource)(nil), ISCSI:(*v1.ISCSIVolumeSource)(nil), Cinder:(*v1.CinderVolumeSource)(nil), CephFS:(*v1.CephFSPersistentVolumeSource)(nil), FC:(*v1.FCVolumeSource)(nil), Flocker:(*v1.FlockerVolumeSource)(nil), FlexVolume:(*v1.FlexVolumeSource)(nil), AzureFile:(*v1.AzureFilePersistentVolumeSource)(nil), VsphereVolume:(*v1.VsphereVirtualDiskVolumeSource)(nil), Quobyte:(*v1.QuobyteVolumeSource)(nil), AzureDisk:(*v1.AzureDiskVolumeSource)(nil), PhotonPersistentDisk:(*v1.PhotonPersistentDiskVolumeSource)(nil), PortworxVolume:(*v1.PortworxVolumeSource)(0xc420414d50), ScaleIO:(*v1.ScaleIOPersistentVolumeSource)(nil), Local:(*v1.LocalVolumeSource)(nil), StorageOS:(*v1.StorageOSPersistentVolumeSource)(nil)}, AccessModes:[]v1.PersistentVolumeAccessMode{"ReadWriteOnce"}, ClaimRef:(*v1.ObjectReference)(0xc4201796c0), PersistentVolumeReclaimPolicy:"Delete", StorageClassName:"ext4-2x-low", MountOptions:[]string(nil)}, err: Feature upgrade needed. Licensed maximum reached for 'Snapshots' feature (allowed 64, requested 65)
E0309 19:35:56.433771       1 goroutinemap.go:166] Operation for "creategitlab/review-bump-10-5-ob7wul-redismillions-redis" failed. No retries permitted until 2018-03-09 19:35:56.933746634 +0000 UTC (durationBeforeRetry 500ms). Error: Failed to take snapshot of the volume pvc-e0fc50db-8d3d-11e7-b368-000d3a035c62: %!q(<nil>)
E0309 19:35:57.010331       1 snapshotter.go:299] No conditions for this snapshot yet.
I0309 19:35:57.015588       1 snapshotter.go:448] findSnapshot: snapshot review-bump-10-5-ob7wul-redis
E0309 19:35:57.015616       1 snapshotter.go:455] Failed to get volume plugin. Snapshot review-bump-10-5-ob7wul-redis is malformed
I0309 19:35:57.015628       1 snapshotter.go:439] syncSnapshot: Creating snapshot gitlab/review-bump-10-5-ob7wul-redis ...
I0309 19:35:57.015635       1 snapshotter.go:481] createSnapshot: Creating snapshot gitlab/review-bump-10-5-ob7wul-redis through the plugin ...
I0309 19:35:57.025924       1 snapshotter.go:487] createSnapshot: Creating metadata for snapshot gitlab/review-bump-10-5-ob7wul-redis.
I0309 19:35:57.025950       1 snapshotter.go:690] In updateVolumeSnapshotMetadata

- apiVersion: volumesnapshot.external-storage.k8s.io/v1
  kind: VolumeSnapshot
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"volumesnapshot.external-storage.k8s.io/v1","kind":"VolumeSnapshot","metadata":{"annotations":{},"name":"review-bump-10-5-ob7wul-redis","namespace":"gitlab"},"spec":{"persistentVolumeClaimName":"millions-redis"}}
    clusterName: ""
    creationTimestamp: 2018-03-09T19:34:57Z
    generation: 0
    initializers: null
    labels:
      SnapshotMetadata-PVName: pvc-e0fc50db-8d3d-11e7-b368-000d3a035c62
      SnapshotMetadata-Timestamp: "1520624221313407023"
    name: review-bump-10-5-ob7wul-redis
    namespace: gitlab
    resourceVersion: "246723736"
    selfLink: /apis/volumesnapshot.external-storage.k8s.io/v1/namespaces/gitlab/volumesnapshots/review-bump-10-5-ob7wul-redis
    uid: f3463bc3-23d0-11e8-b26a-000d3a031a10
  spec:
    persistentVolumeClaimName: millions-redis
    snapshotDataName: ""
  status:
    conditions: null
    creationTimestamp: null
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    pv.kubernetes.io/bind-completed: "yes"
    pv.kubernetes.io/bound-by-controller: "yes"
    volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/portworx-volume
  creationTimestamp: 2017-08-30T04:44:16Z
  name: millions-redis
  namespace: gitlab
  resourceVersion: "168985060"
  selfLink: /api/v1/namespaces/gitlab/persistentvolumeclaims/millions-redis
  uid: e0fc50db-8d3d-11e7-b368-000d3a035c62
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: ext4-2x-low
  volumeName: pvc-e0fc50db-8d3d-11e7-b368-000d3a035c62
status:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 10Gi
  phase: Bound
apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    kubernetes.io/createdby: portworx-volume-dynamic-provisioner
    pv.kubernetes.io/bound-by-controller: "yes"
    pv.kubernetes.io/provisioned-by: kubernetes.io/portworx-volume
  creationTimestamp: 2017-08-30T04:44:20Z
  name: pvc-e0fc50db-8d3d-11e7-b368-000d3a035c62
  resourceVersion: "168985056"
  selfLink: /api/v1/persistentvolumes/pvc-e0fc50db-8d3d-11e7-b368-000d3a035c62
  uid: e32af46f-8d3d-11e7-b368-000d3a035c62
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 10Gi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: millions-redis
    namespace: gitlab
    resourceVersion: "168985004"
    uid: e0fc50db-8d3d-11e7-b368-000d3a035c62
  persistentVolumeReclaimPolicy: Delete
  portworxVolume:
    volumeID: "805925391547708106"
  storageClassName: ext4-2x-low
status:
  phase: Bound
cehoffman commented

I think I know where this is coming from. This is related to the question I had in slack yesterday about the snapshot limit.

disrani-px commented

Those are non-fatal errors. Should be fixed with changes in this PR: kubernetes-retired/external-storage#647