Adding PVC's annotation to CreateVolumeRequest.Parameters
orainxiong opened this issue Β· 65 comments
As the title says, there is a great possibility that the application-specific Operator would leverage PVC's Annotation for storing custom data. The custom data will tell storage provisioner how CSI drive create volume correctly.
Here is already existing logic for provision volume :
// Create a CSI CreateVolumeRequest and Response
req := csi.CreateVolumeRequest{
Name: pvName,
Parameters: options.Parameters,
VolumeCapabilities: []*csi.VolumeCapability{
{
AccessType: accessType,
AccessMode: accessMode,
},
},
CapacityRange: &csi.CapacityRange{
RequiredBytes: int64(volSizeBytes),
},
}
Note : more details
CSI Plugin specifies that CreateVolumeRequest's field Parameters is OPTIONAL and opaque.
What I am suggesting is that we could pass in PVC's Annotation in addition to Storageclass.Parameter.
I second this request, and we discussed this idea with @saad-ali at this week's Storage SIG F2F. The chief concern is name collisions between storage class parameters and PVC annotations. Saad suggested this capability be gated via a startup flag. @orainxiong, are you planning to submit a PR for this?
@clintonk Thx for your replay.
In order to solve this issue immediately, I handle this issue in a temporary way.
- A new function, called
appendPVCAnnotation, is introduced to encode PVC annotation to JSON string and modifyoptions.parameter.
func appendPVCAnnotation(options controller.VolumeOptions) error {
annotationPVC, err := json.Marshal(options.PVC.Annotations)
if err != nil {
return err
}
parameters := map[string]string{
"annotationsPVC": string(annotation),
}
return nil
}
Note : Reverse key name annotationsPVC to avoid name conflict.
- A new flag, called
annotationPVCNeeded, is added to determine whether PVC annotation should be appended intooptions.parameter.
The advantage of implementation hardly modifies existing external-provisioner logic. Honestly, it is a not a preferable way.
Looking forward to your suggestions.
While I agree that there should be a way how to pass parameters from user to CreateVolume, I don't like suggested solution:
-
It makes the CSI driver Kubernetes aware. Whole point of CSI is to have independent plugins that can be used in Kubernetes, Mesos, Cloud Foundry, Docker Swarm and others. The plugins should not know anything about PVCs, these are pure Kubernetes objects.
-
Kubernetes admin needs a way how to restrict what parameters can be set by users. Kubernetes is (or can be) multi tenant system and not all users can use all parameters that a CSI plugin exposes.
-
User should be allowed to list possible parameters together with some description so they can choose the right value. Note that there may be GUI between users and Kubernetes, so the GUI must be able at least to enumerate these parameters. We tried in kubernetes/community#543, however, nobody has stepped in to finish the proposal and implemented it.
After creating the csi-digitalocean driver, I also received a feedback, that needs this feature: digitalocean/csi-digitalocean#30
I'm using the name of the request ot create the volumes, which ends up being the PV name. So it's in the form of kubernetes-dynamic-pv-<randomid>. However on the UI (at DigitalOcean.com), this is not useful because it doesn't give the person an easy way to track back to which entity it belongs to.
I thought maybe I can use the StorageClass parameters and add a new field called volumePrefix, such as:
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: do-block-storage
namespace: kube-system
annotations:
storageclass.kubernetes.io/is-default-class: "true"
provisioner: com.digitalocean.csi.dobs
parameters:
volumePrefix: "csi-digitalocean"
and then in the Controller plugin, I'll change the name in CreateVolume according to this information:
volumeName := req.Name
// change the name if the storage-class has a prefix option
prefix, ok := req.Parameters["volumePrefix"]
if ok {
volumeName = prefix + "-" + volumeName
}This allows me to create a volume based on this users CSI deployment. However, this is not scalable because:
- you only create a single StorageClass, so it's not easy to update it every single time
- you're not allows to update the parameters field of a StorageClass
- StorageClass is global, whereas we need a local information from a local resource
A better idea would be to use the annotations of a PVC. This would allow us to create volume's directly based on the information bound to a specific PVC.
We're already passing down the information from a Kubernetes resource (i.e StorageClass) down to the plugin, I think there is not much that prevents us to pass the information fo another resource (i.e PersistentVolumeClaim) to the plugin.
check this out:
https://github.com/kubernetes-csi/external-provisioner/blob/master/cmd/csi-provisioner/csi-provisioner.go#L46
I think you can leverage this parameter.
@sbezverk thanks for the tip, I didn't know it.
Though, the prefix was just an example, it could be also something else. Such tagging the resource on the cloud provider side based on the created PVC labels.
@jsafrane
Thx for your reply.
If I understand correctly, I notice that external-provisioner has exposed StorageClass parameters to end storage drivers through VolumeOptions.
options := VolumeOptions{
PersistentVolumeReclaimPolicy: reclaimPolicy,
PVName: pvName,
PVC: claim,
Parameters: parameters, -- Store `StorageClass` parameters here
}
volume, err = ctrl.provisioner.Provision(options)
End storage drivers can get StorageClass parameters depending on their own logic. However, PVC's annotation is still missing in VolumeOptions.
I very agree with your opinion. It's a temporary solution which makes storage driver Kubernetes aware. It is a very time consuming job to figure out a COs neutral solution.
As far as I think of parameters and description, it is very likely to be exhaust to create a common body of naming language across COs.
Looking forward to your suggestions.
@fatih, for tagging we could perhaps extend CSI's CreateVolume call with optional metadata such as Kubernetes PVC namespace + name, PV name, storage class name (?) or PVC labels (which one? all of them?). It would be used only to tag volumes in the storage backend so users can map these volumes to CO objects. These cannot be used to configure provisioned volume (see #86 (comment) above).
CO would provide generic map[string]string, opaque to CSI driver and CSI driver would blindly "attach" it to the volume via AWS tags or GCE labels or OpenStack metadata or whatever the storage supports (or throw them away if the storage does not support tagging).
However, PVC's annotation is still missing in VolumeOptions.
@orainxiong, that's intentional.
I could imagine our CSI provisioner passes content of some alpha annotations from PVC as CreateVolumeRequest.parameters in CreateVolume call, but these should be really alpha. To get the to GA, we need policy and discovery and whatnot.
for tagging we could perhaps extend CSI's CreateVolume call with optional metadata such as Kubernetes PVC namespace + name, PV name, storage class name (?) or PVC labels (which one? all of them?).
I think passing the PVC metadata would be great. Because it already includes the following important bits:
- name
- namespace
- labels
- annotations
I'm not sure how to pass them via CO though. a map[string]string is the easiest way. But then how are we going to represent nested maps? We could pass it as a JSON maybe:
map[string]string{
"pvc" : "{\"name\": ...}",
"storageclass": " { ... }",
}
I know that Kubernetes annotations mostly store data like this and it's a common thing there.
CO would provide generic map[string]string, opaque to CSI driver and CSI driver would blindly "attach" it to the volume via AWS tags or GCE labels or OpenStack metadata or whatever the storage supports (or throw them away if the storage does not support tagging).
Yeap. Attaching is only one of the things the CSI driver can accomplish. Changing the name is another action it should be allowed to do.
@jsafrane Thanks. I got it. I will close this issue.
Why is this closed? Did we have a PR that fixes this issue or a solution that is applicable right now?
At least, in my opinion, there is a temporary way to handle this issue. But, it isn't applicable for all COs that support CSI.
As your suggestions, a new function injectCOParameter is introduced into external-provisioner :
func injectCOParameter(options controller.VolumeOptions) (map[string]string, error) {
annotationsPVC, err := json.Marshal(options.PVC.Annotations)
if err != nil {
return nil, err
}
labelsPVC, err := json.Marshal(options.PVC.Labels)
if err != nil {
return nil, err
}
storageClass, err := json.Marshal(options.Parameters)
if err != nil {
return nil, err
}
parameters := map[string]string{
"annotationsPVC": string(annotationsPVC),
"labelsPVC": string(labelsPVC),
"storageClass": string(storageClass),
}
return parameters, nil
}
injectCOParameter is in the charge of encoding labels and annotations and storageclass into
CreateVolumeRequest.Parameters. Specifically, CreateVolumeRequest.Parameters is defined by CSI to store opaque parameter and COs will ignore it.
On the storage provisioner side, they can decode parameters from CreateVolumeRequest.Parameters according to their own operational logic.
@orainxiong Thanks for the reply. The title says Adding PVC's annotation to CreateVolumeRequest.Parameters and unless I'm missing something, there is no such feature and no one contributed to the provisioner/attacher to add this feature. Hence my question why we have closed this issue. Nothing is fixed yet and it's not possible to pass any PVC/PV data to the plugin from the CO. This issue should be re-opened, and closed when there is an action task we can work together.
@fatih ok, you're right. : - )
Has injectCOParameter() already been merged to external-provisioner?
I couldn't find it in the codes nor PRs, so I would appreciate it if you could share any links to it.
@mkimuram
Hope it works for you.
https://github.com/orainxiong/external-provisioner/tree/issue-86
@orainxiong
Thank you for providing code. Great work!
I tested your code as below and confirmed that annotationsPVC and labelsPVC were added to parameters that are passed to each driver.
The concept looks good to me and will solve my concern(#93).
In my opinion, there would be still rooms for discussion to make this code merged:
- What parameters will be needed to be passed,
(I found that annotation for storage class is not passed, and there might be some other fields that will be useful if we pass.) - How they will be passed,
(Should parameters for storageClass be passed directory to keep compatibility, and so on?) - Should they be passed in the first place,
(Although, I personally believe that they should, it doesn't seem to be agreed in the community.)
I found a related discussion below for 3, and this will be needed to be discussed, first .
container-storage-interface/spec#248
Any comments for this issue? @jsafrane @princerachit @vladimirvivien
Especially, I would like to know whether k8s can add this code as a temporary workaround, before we reach to a conclusion for above CSI discussion.
Summary of test Results:
[Without patch]
# kubectl logs csi-pod hostpath-driver
I0612 18:57:50.339548 1 utils.go:97] GRPC request: name:"pvc-80ee79686e7211e8" capacity_range:<required_bytes:1073741824 > volume_capabilities:<mount:<> access_mode:<mode:SINGLE_NODE_WRITER > > parameters:<key:"foo21" value:"bar21" > parameters:<key:"foo22" value:"bar22" >
[With patch]
# kubectl logs csi-pod hostpath-driver
I0612 20:37:56.738211 1 utils.go:97] GRPC request: name:"pvc-7d0658ed6e8011e8" capacity_range:<required_bytes:1073741824 > volume_capabilities:<mount:<> access_mode:<mode:SINGLE_NODE_WRITER > > parameters:<key:"annotationsPVC" value:"{\"foo31\":\"bar31\",\"foo32\":\"bar32\",\"volume.beta.kubernetes.io/storage-provisioner\":\"csi-hostpath\"}" > parameters:<key:"labelsPVC" value:"{\"foo41\":\"bar41\",\"foo42\":\"bar42\"}" > parameters:<key:"storageClass" value:"{\"foo21\":\"bar21\",\"foo22\":\"bar22\"}" >
Please see below for how I tested:
[Without patch] (Also see [files] below for csi-sc2.yaml and csi-pvc2.yaml.)
# FEATURE_GATES=CSIPersistentVolume=true ALLOW_PRIVILEGED=true ALLOW_SECURITY_CONTEXT=true hack/local-up-cluster.sh
# export KUBECONFIG=/var/run/kubernetes/admin.kubeconfig
# kubectl create -f https://raw.githubusercontent.com/kubernetes-csi/docs/master/book/src/example/csi-setup.yaml
# kubectl create -f csi-sc2.yaml
# kubectl create -f csi-pvc2.yaml
# kubectl logs csi-pod hostpath-driver | grep -A 3 "CreateVolume" | tail -4
I0612 18:57:50.339545 1 utils.go:96] GRPC call: /csi.v0.Controller/CreateVolume
I0612 18:57:50.339548 1 utils.go:97] GRPC request: name:"pvc-80ee79686e7211e8" capacity_range:<required_bytes:1073741824 > volume_capabilities:<mount:<> access_mode:<mode:SINGLE_NODE_WRITER > > parameters:<key:"foo21" value:"bar21" > parameters:<key:"foo22" value:"bar22" >
I0612 18:57:50.339719 1 controllerserver.go:87] create volume /tmp/80f0db83-6e72-11e8-b5be-0242ac110003
I0612 18:57:50.339724 1 utils.go:102] GRPC response: volume:<capacity_bytes:1073741824 id:"80f0db83-6e72-11e8-b5be-0242ac110003" attributes:<key:"foo21" value:"bar21" > attributes:<key:"foo22" value:"bar22" > >
[With patch] (Also see [files] below for csi-sc2.yaml and csi-pvc2.yaml.)
# git clone https://github.com/orainxiong/external-provisioner.git
# cd external-provisioner
# git checkout issue-86
# sed -i "s/canary/hack/" Makefile
# make container
# curl https://raw.githubusercontent.com/kubernetes-csi/docs/master/book/src/example/csi-setup.yaml | sed "s/quay.io\/k8scsi\/csi-provisioner:v0.2.1/quay.io\/k8scsi\/csi-provisioner:hack/" > csi-setup2.yaml
# sed -i "s/imagePullPolicy: Always/imagePullPolicy: Never/" csi-setup2.yaml
# FEATURE_GATES=CSIPersistentVolume=true ALLOW_PRIVILEGED=true ALLOW_SECURITY_CONTEXT=true hack/local-up-cluster.sh
# export KUBECONFIG=/var/run/kubernetes/admin.kubeconfig
# kubectl create -f csi-setup2.yaml
# kubectl create -f csi-sc2.yaml
# kubectl create -f csi-pvc2.yaml
# kubectl logs csi-pod hostpath-driver | grep -A 3 "CreateVolume" | tail -4
I0612 20:37:56.738207 1 utils.go:96] GRPC call: /csi.v0.Controller/CreateVolume
I0612 20:37:56.738211 1 utils.go:97] GRPC request: name:"pvc-7d0658ed6e8011e8" capacity_range:<required_bytes:1073741824 > volume_capabilities:<mount:<> access_mode:<mode:SINGLE_NODE_WRITER > > parameters:<key:"annotationsPVC" value:"{\"foo31\":\"bar31\",\"foo32\":\"bar32\",\"volume.beta.kubernetes.io/storage-provisioner\":\"csi-hostpath\"}" > parameters:<key:"labelsPVC" value:"{\"foo41\":\"bar41\",\"foo42\":\"bar42\"}" > parameters:<key:"storageClass" value:"{\"foo21\":\"bar21\",\"foo22\":\"bar22\"}" >
I0612 20:37:56.738409 1 controllerserver.go:87] create volume /tmp/7d088fc3-6e80-11e8-84a2-0242ac110002
I0612 20:37:56.738416 1 utils.go:102] GRPC response: volume:<capacity_bytes:1073741824 id:"7d088fc3-6e80-11e8-84a2-0242ac110002" attributes:<key:"annotationsPVC" value:"{\"foo31\":\"bar31\",\"foo32\":\"bar32\",\"volume.beta.kubernetes.io/storage-provisioner\":\"csi-hostpath\"}" > attributes:<key:"labelsPVC" value:"{\"foo41\":\"bar41\",\"foo42\":\"bar42\"}" > attributes:<key:"storageClass" value:"{\"foo21\":\"bar21\",\"foo22\":\"bar22\"}" > >
# kubectl logs csi-pod external-provisioner | grep -A 3 "CreateVolume" | tail -4
I0612 20:37:56.738068 1 controller.go:104] GRPC call: /csi.v0.Controller/CreateVolume
I0612 20:37:56.738072 1 controller.go:105] GRPC request: name:"pvc-7d0658ed6e8011e8" capacity_range:<required_bytes:1073741824 > volume_capabilities:<mount:<> access_mode:<mode:SINGLE_NODE_WRITER > > parameters:<key:"annotationsPVC" value:"{\"foo31\":\"bar31\",\"foo32\":\"bar32\",\"volume.beta.kubernetes.io/storage-provisioner\":\"csi-hostpath\"}" > parameters:<key:"labelsPVC" value:"{\"foo41\":\"bar41\",\"foo42\":\"bar42\"}" > parameters:<key:"storageClass" value:"{\"foo21\":\"bar21\",\"foo22\":\"bar22\"}" >
I0612 20:37:56.738515 1 controller.go:107] GRPC response: volume:<capacity_bytes:1073741824 id:"7d088fc3-6e80-11e8-84a2-0242ac110002" attributes:<key:"annotationsPVC" value:"{\"foo31\":\"bar31\",\"foo32\":\"bar32\",\"volume.beta.kubernetes.io/storage-provisioner\":\"csi-hostpath\"}" > attributes:<key:"labelsPVC" value:"{\"foo41\":\"bar41\",\"foo42\":\"bar42\"}" > attributes:<key:"storageClass" value:"{\"foo21\":\"bar21\",\"foo22\":\"bar22\"}" > >
I0612 20:37:56.738538 1 controller.go:108] GRPC error: <nil>
[Files]
# cat csi-sc2.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: csi-hostpath-sc2
annotations:
foo11: bar11
foo12: bar12
provisioner: csi-hostpath
reclaimPolicy: Delete
volumeBindingMode: Immediate
parameters:
foo21: bar21
foo22: bar22
# cat csi-pvc2.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: csi-pvc2
annotations:
foo31: bar31
foo32: bar32
labels:
foo41: bar41
foo42: bar42
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
storageClassName: csi-hostpath-sc2
@mkimuram today's discussion concluded that we are not going to immediately touch the spec for CreateVolumeRequest to pass CO specific metadata(PVC name,PVC id etc). A proposed workaround is to write your own external provisioner which knows what params to pass for your specific driver.
However, as pvc annotations are also used to add volume related information, adding them to external-provisioner would be possible. We need someone from k8s wg-csi to nod on this. @saad-ali
@clintonk You're welcome. : - )
We've talked about this a bit in kubernetes/community#2273 as well.
I'd like a way to have VolumeAttributes be passed in PVC's and the external provisioner can decide if its ok merging none/some/all of them with the StorageClasses VolumeAttributes when sending to the CSI drivers Create/DeleteVolume. And also if none/some/all should be copied to the resulting PV for the CSI driver's mount/unmount.
This wouldn't be specific to Kubernetes, as they would be CSI attributes, not Kubernetes annotations.
I'd like a way to have VolumeAttributes be passed in PVC's and the external provisioner can decide if its ok merging none/some/all of them with the StorageClasses VolumeAttributes when sending to the CSI drivers Create/DeleteVolume.
Is it the same logic to @orainxiong 's above code or different one?
According to @princerachit 's below comment, at least pvc annotations could be allowed to be passed by above way, if k8s wg-csi agrees.
However, as pvc annotations are also used to add volume related information, adding them to external-provisioner would be possible. We need someone from k8s wg-csi to nod on this. @saad-ali
My current standing on this issue: Annotations on PVCs MUST NOT be passed to CSI drivers. The Kubernetes PVC object is intended for application portability. When we start leaking cluster/implementation specific details in to it we are violating that principle. And explicitly passing PVC annotations to CSI drivers encourages that pattern. Workload portability is a corner stone of Kubernetes. It is one of the major reasons for the success of the project. We must be very careful when we violate it.
Instead let's focus on specific use cases. And let's see if we can come up with better solutions for each of those uses cases rather then opening up a hole in the API.
@saad-ali There is a similar use case in my team:
We support multiple storage types in a cluster, such as Ceph RBD and CephFS. What's more, we also support multiple file system types for Ceph RBD, such as ext4, xfs, etc. The only solution at present is creating a StorageClass for each file system type, for example, a ext4-sc StorageClass for ext4 fs type, and a xfs-sc StorageClass for xfs fs type.
However, we also need to set quota for PVCs, so we need to set quota for these two StorageClasses(ext4-sc and xfs-sc) separately if we adopt this approach. This is strange for users because these two StorageClasses are the same storage type(Ceph RBD), just different file systems.
StorageClass object was supposed to provide classes of storage ("slow", "fast", "secure"), not configuration of storage ("ssd-4-replicas-xfs", "ssd-4-replicas-ext4", "ssd-3-replicas-xfs"...)
Why should user need to choose between say ext2 or xfs? Users need storage to run say "fast cache", where ext2 simplicity and speed is better than xfs or "database XYZ", where xfs is more stable than ext2 and XYZ is more optimized for xfs than ext4 (all examples are completely fictional).
So there should be a class "fast cache", "database XYZ" and "default". How many such classes do you need in a cluster? Maybe lot of them, but their number grows with number of applications and their different needs, not with number of configuration options each storage provides.
We have similar situation.
@jsafrane The word "user" today means too many things. Do you mean "the end user" which equals to DevOps or SRE, or "PaaS user" which build their ability on the k8s?
We are the latter one who focus on internal operations of a company, and yes the end user do not care about ext4 or xfs, but we care. We do not expose the whole k8s abstraction to the end user. So we need detailed customization rather than several wild classes.
How portworx openstorage get pvc annotations in csi driver?
pvc:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: csi-pvc
annotations:
openstorage.io/auth-secret-name: example-secret-name
openstorage.io/auth-secret-namespace: example-secret-namespace
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
storageClassName: storage-class csi controller server:
func (s *OsdCsiServer) CreateVolume(
ctx context.Context,
req *csi.CreateVolumeRequest,
) (*csi.CreateVolumeResponse, error) {
// Get parameters
spec, locator, source, err := s.specHandler.SpecFromOpts(req.GetParameters())
if err != nil {
e := fmt.Sprintf("Unable to get parameters: %s\n", err.Error())
logrus.Errorln(e)
return nil, status.Error(codes.InvalidArgument, e)
}
}openstorage server:
spec := dcReq.GetSpec()
locator := dcReq.GetLocator()
secretName, secretContext, err := a.parseSecret(spec.VolumeLabels, locator.VolumeLabels, true)How portworx openstorage get pvc annotations in csi driver?
I think they use a fork of external-provisioner for this. @lpabon might be able to confirm that.
@kerneltime mentioned another external-provisioner fork here: #170 (comment)
My take on this: when multiple people come to the conclusion that they need to fork external-provisioner to implement their use cases, then this is an indication that something is missing in what is provided upstream. We might want to reconsider that...
StorageClass object was supposed to provide classes of storage ("slow", "fast", "secure"), not configuration of storage ("ssd-4-replicas-xfs", "ssd-4-replicas-ext4", "ssd-3-replicas-xfs"...)
Why should user need to choose between say ext2 or xfs? Users need storage to run say "fast cache", where ext2 simplicity and speed is better than xfs or "database XYZ", where xfs is more stable than ext2 and XYZ is more optimized for xfs than ext4 (all examples are completely fictional).
So there should be a class "fast cache", "database XYZ" and "default". How many such classes do you need in a cluster? Maybe lot of them, but their number grows with number of applications and their different needs, not with number of configuration options each storage provides.
@jsafrane This argument does not work for the enterprise. The problem of how to use a filesystem efficiently for any given application is complex, and discerning enterprise admins know more about their application than any orchestrator/storage vendor can.
Not only is the choice of filesystem important, but to make matters worse, each filesystem has tradeoffs between integrity and performance. We have seen customers solve this problem on a per application basis by setting custom mount options.
For e.g ordered-data mode (journaling mode) provides better integrity of the ext-x family of filesystems, but of course it is slower. Some db vendors recommend integrity settings for their log volumes, but performance-biased settings for their data volumes.
Long story short, I think there is a need for per-volume, volume-type-specific parameters/configuration that enterprise users will want. I agree storage classes are not the right tool to use it. I also agree with @saad-ali that annotations break abstractions.
If the community agrees about NOT supporting annotations, that decision should be accompanied by an alternative approach. Otherwise, the CSI API falls short of expectations.
My suggestion : A sidecar release that supports annotations as "deprecated parameters" could serve as a transition path. That will prevent forks of sidecars (which is a big win for the end user).
The problem then becomes the discussion of which use-cases should be promoted to first class CSI API ..... and also how to end-of-life this sidecar.
Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale
/remove-lifecycle stale
My current standing on this issue: Annotations on PVCs MUST NOT be passed to CSI drivers. The Kubernetes PVC object is intended for application portability. When we start leaking cluster/implementation specific details in to it we are violating that principle. And explicitly passing PVC annotations to CSI drivers encourages that pattern. Workload portability is a corner stone of Kubernetes. It is one of the major reasons for the success of the project. We must be very careful when we violate it.
Instead let's focus on specific use cases. And let's see if we can come up with better solutions for each of those uses cases rather then opening up a hole in the API.
Our use case at Ticketmaster is tagging EBS volumes when we create PVCs. We use tags to associate costs and we would need a way to pass additional tags to PVCs. Our current workaround is a controller that watches PVCs and based on an annotation it tags the volumes.
I do not see any difference between this use case and the use case of creating a load balancer and adding an annotation to tag the resource:
service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags: "Tag1=value1,Tag2=value2"
This is creating a resource in AWS, the same the CSI driver does when creating a PVC, and in the case of the LoadBalancer there is an annotation to handle tags. Why can't we have the same user experience?
@saad-ali Our use case is to add the details pf PVC (pvc name, Uuid) as metadata to resource created on storage array. is there anyway to achieve this as part of create request.
Initially in external storage we were able to grab these details.
We would like to allow customers to "import" an existing volume as a PVC, something similar to OpenStack cinder manage functionality.
Think about a volume created as a replica from another storage array, or restored from some external backup device, or created with older dynamic volume provisioner - outside of the CSI driver.
@saad-ali - If the annotations are not supported, what would be the best way to transfer the actual volume name to the CSI driver?
@gregnsk, there is #399, where a driver can optionally get PV/PVC name. My use case is to "tag" / "label" the volume so if Kubernetes dies from whatever reason, storage admin still can find which volume is what.
But the driver can do anything with this information and we can't prevent it reading PVC annotations from Kubernetes API server. *cough* *cough*
@gregnsk for your use case, you can manually create a PV object to import a pre-existing volume
What kind of management are you looking for? Once you manually create a PV and bind it to a PVC, you can use it like any other dynamically provisioned volume
expand/clone/snapshot/delete. All CSI functionality.
All these operations can be done for manually created PVs as long as you associate them with a storageclass
interesting. how can I associate a manually created PV with a storage class?
PV.spec.storageclassname. And also set the same storageclassname on the PVC.
We also need to import an existing volume with CSI, what's the timeline for next release of external provisioner to take PVC name in CSI (#399 )
@haibinxie you don't need #399 to import an existing volume with CSI. This is already supported through manually creating a PV.
@haibinxie @gregnsk here's an example driver documentation on how to manually create a PV for an existing disk: https://github.com/kubernetes-sigs/gcp-compute-persistent-disk-csi-driver/blob/master/docs/kubernetes/user-guides/pre-provisioned.md
@msau42 Thanks for sharing the info, in that example how do you guarantee the PV would not be bound to other PVC before you create PVC, i have seen others set claimRef in PV to ensure that.
Yes setting PV.ClaimRef is the recommended way
@msau42 is volumeHandle in PV object gets passed to CSI CreateVolumeRequest.Name so CSI driver just needs to check whether the volume exists or not, and import if it exists.
if that's the case can we also fill it somewhere in PVC to be carried to CreateVolumeRequest.name?
@haibinxie if you manually create the PV then CreateVolume does not get invoked. Can we open another issue to discuss this import case? This is veering off topic from the original issue.
Has there been any update on implementing this feature? We would greatly appreciate it.
cc @saad-ali
We also use tags to calculate the cost of each environment.
Therefore we need a way to tag AWS volumes created by CSI external provisioner.
Until there will be a proper way to define the tags, I use this script to go over all un-tagged volumes and tag them according to the attached instance tag:
https://gist.github.com/eladtamary/49de1406e2605d2d659cbcd0b420eba1
Maybe it will help someone else.
Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale
/remove-lifecycle stale
/lifecycle frozen
This argument extra-create-metadata passes pvc ns/name to CreateVolumeRequest.Parameters. This is really useful.
I need to map existing shares of NAS filers to PV without the cluster/storage admins intervention #kubernetes-csi/csi-driver-smb#398 hence placing the sharename in the PVC would be the solution to achieve this.
If using an annotation is not suitable since it would break the CSI specs, would it be possible to use an extra parameter as described in #714 to pass parameters directly to the driver?
This would also help to tag Azure file share/disk by overriding parameters.tags of StorageClass based on Azure CSI drivers
Interestingly, there are CSI drivers which apparently provide support for PVC annotations to override SC parameters:
https://scod.hpedev.io/csi_driver/using.html#using_pvc_overrides
since passing pvc annotation to CreateVolumeRequest.Parameters is not supported by external-provisioner, I see there is other project(e.g. HPE CSI Driver) trying to cache all pvc and then pass pvc annotation to the CSI driver, not a graceful implementation, but looks like this is the only way now:
https://github.com/hpe-storage/csi-driver/blob/abf94a40b242477e51afac75bb9a68772aede0e7/pkg/flavor/kubernetes/flavor.go#L163-L172
We're also in need of a feature like this to be able to associate disks with applications to better track costs.