failed to find the sidecar container in Pod spec

Question

failed to find the sidecar container in Pod spec

DattatrayHinge opened this issue a year ago · 16 comments

Hi
I am trying to use examples/static/pv-pvc-deployment.yaml
but when I describe the pod getting below error

MountVolume.SetUp failed for volume "gcs-fuse-csi-pv" : rpc error: code = FailedPrecondition desc = failed to find the sidecar container in Pod spec

Answer 1 · 2023-12-27T14:42:27.000Z

Hi @DattatrayHinge , could you check if you put the Pod annotation gke-gcsfuse/volumes: "true"?

Answer 2 · 2024-01-05T15:28:33.000Z

Hi Jiaxun Thanks for email. it was disabled before and now it is enabled but then a new error pop up in events annotations:

gke-gcsfuse/volumes: "true"

Errors in event Events:

Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedScheduling 30s (x2 over 33s) gke.io/optimize-utilization-scheduler 0/3 nodes are available: 3 Insufficient ephemeral-storage. preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod..

Normal NotTriggerScaleUp 30s cluster-autoscaler

…

pod didn't trigger scale-up: 3 Insufficient ephemeral-storage On Wed, 27 Dec 2023 at 20:12, Jiaxun Song ***@***.***> wrote: Hi @DattatrayHinge <https://github.com/DattatrayHinge> , could you check if you put the Pod annotation gke-gcsfuse/volumes: "true"? — Reply to this email directly, view it on GitHub <#153 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AE24TIM4HS2FDDGRTJG67MTYLQXV3AVCNFSM6AAAAABBDLOBXGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNZQGM3DQNRVGY> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- Best Regards, Dattatray Hinge +919987658128

Answer 3 · 2024-01-11T03:37:11.000Z

Hi @DattatrayHinge , please use the annotation gke-gcsfuse/ephemeral-storage-limit: 1Gi to lower the sidecar container ephemeral storage and retry. See https://cloud.google.com/kubernetes-engine/docs/how-to/persistent-volumes/cloud-storage-fuse-csi-driver#sidecar-container for details

Answer 4 · 2024-01-19T15:25:19.000Z

Thanks Jiaxun, It is working now, but I am getting permission errors on error creating fragment file (13: Permission denied) Is it like I have to set some permissions in sidecar container?

…

On Thu, 11 Jan 2024 at 09:07, Jiaxun Song ***@***.***> wrote: Hi @DattatrayHinge <https://github.com/DattatrayHinge> , please use the annotation gke-gcsfuse/ephemeral-storage-limit: 1Gi to lower the sidecar container ephemeral storage and retry. See https://cloud.google.com/kubernetes-engine/docs/how-to/persistent-volumes/cloud-storage-fuse-csi-driver#sidecar-container for details — Reply to this email directly, view it on GitHub <#153 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AE24TILDAU2QUHK7OGE7VPTYN5M7DAVCNFSM6AAAAABBDLOBXGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQOBWGE3TEMJWGA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- Best Regards, Dattatray Hinge +919987658128

Answer 5 · 2024-01-19T18:43:12.000Z

Hi @DattatrayHinge ,

Yes, please find the solution on the GKE doc:

If you use a Security Context for your Pod or container, or if your container image uses a non-root user or group, you must set the uid and gid mount flags. You also need to use the file-mode and dir-mode mount flags to set the file system permissions. Note that you cannot run chmod, chown, or chgrp commands against a Cloud Storage FUSE file system, so uid, gid, file-mode and dir-mode mount flags are necessary to provide access to a non-root user or group.

Answer 6 · 2024-01-20T20:30:50.000Z

That's great help Jiaxun This resolved my problem, Thank you very much.

…

On Sat, 20 Jan 2024 at 00:13, Jiaxun Song ***@***.***> wrote: Hi @DattatrayHinge <https://github.com/DattatrayHinge> , Yes, please find the solution on the GKE doc <https://cloud.google.com/kubernetes-engine/docs/how-to/persistent-volumes/cloud-storage-fuse-csi-driver#mounting-flags> : If you use a Security Context <https://cloud.google.com/kubernetes-engine/docs/how-to/persistent-volumes/(https:/kubernetes.io/docs/tasks/configure-pod-container/security-context/)%7B:.external%20track-name=%22k8sLink%22%20track-type=%22concept%22%7D> for your Pod or container, or if your container image uses a non-root user or group, you must set the uid and gid mount flags. You also need to use the file-mode and dir-mode mount flags to set the file system permissions. Note that you cannot run chmod, chown, or chgrp commands against a Cloud Storage FUSE file system, so uid, gid, file-mode and dir-mode mount flags are necessary to provide access to a non-root user or group. — Reply to this email directly, view it on GitHub <#153 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AE24TIMKGC2ROUNWVUXQQTLYPK5EXAVCNFSM6AAAAABBDLOBXGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMBQHEZDIMJUG4> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- Best Regards, Dattatray Hinge +919987658128

Answer 7 · 2024-03-26T06:52:09.000Z

HI @songjiaxun , i am facing the rpc error: code = FailedPrecondition desc = failed to find the sidecar container in Pod spec while implementation GCS-CSI Driver and its working fine with nginx pod but not working with deployment yaml.

Note - annotations already added.

Kindly help here.

Answer 8 · 2024-03-26T15:32:04.000Z

Hi @guru91197 , can you share the deployment yaml? Please note that for deployment yaml, the annotation needs to be added under the spec.template.metadata.annotations field.

Answer 9 · 2024-03-26T19:19:14.000Z

yes, I have the below annotations annotations: gke-gcsfuse/volumes: "true" gke-gcsfuse/ephemeral-storage-limit: 1Gi

…

On Wed, 27 Dec 2023 at 20:12, Jiaxun Song ***@***.***> wrote: Hi @DattatrayHinge <https://github.com/DattatrayHinge> , could you check if you put the Pod annotation gke-gcsfuse/volumes: "true"? — Reply to this email directly, view it on GitHub <#153 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AE24TIM4HS2FDDGRTJG67MTYLQXV3AVCNFSM6AAAAABBDLOBXGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNZQGM3DQNRVGY> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- Best Regards, Dattatray Hinge +919987658128

Answer 10 · 2024-03-26T19:42:50.000Z

Hi @DattatrayHinge, are you facing the same issue as @guru91197 reported?

Can you double check if the annotations are under spec.template.metadata.annotations in a deployment spec? Note that it is not under the deployment annotations, it's under the Pod template annotations.

Answer 11 · 2024-03-26T20:05:06.000Z

This is my deployment yaml [image: image.png]

…

On Wed, 27 Mar 2024 at 01:13, Jiaxun Song ***@***.***> wrote: Hi @DattatrayHinge <https://github.com/DattatrayHinge>, are you facing the same issue as @guru91197 <https://github.com/guru91197> reported? Can you double check if the annotations are under spec.template.metadata.annotations in a deployment spec? Note that it is not under the deployment annotations, it's under the Pod *template* annotations. — Reply to this email directly, view it on GitHub <#153 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AE24TILK4D2PJJWGRONFUDLY2HFVBAVCNFSM6AAAAABBDLOBXGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMRRGMZDKOBVGU> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- Best Regards, Dattatray Hinge +919987658128

Answer 12 · 2024-03-26T20:11:33.000Z

@DattatrayHinge I don't think the yaml is attached.

Answer 13 · 2024-03-26T20:42:12.000Z

[image: image.png]

…

On Wed, 27 Mar 2024 at 01:41, Jiaxun Song ***@***.***> wrote: @DattatrayHinge <https://github.com/DattatrayHinge> I don't think the yaml is attached. — Reply to this email directly, view it on GitHub <#153 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AE24TIO7OEA53NXACHN2V6TY2HJAXAVCNFSM6AAAAABBDLOBXGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMRRGM3TMMBYGU> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- Best Regards, Dattatray Hinge +919987658128

Answer 14 · 2024-03-26T22:29:51.000Z

Hi @DattatrayHinge I guess GitHub does not allow you to attach an image via email. Can you share the spec yaml directly?

Answer 15 · 2024-04-06T05:34:55.000Z

I have been using the driver in existing deployments since Oct 2023 and the annotation was in spec.metadata.annotations. When did it move to spec.template.metadata.annotations? I spent several hours trying to debug a new deployment until I found this thread and moved the annotation to spec.template.metadata.annotations. Now it works!

Answer 16 · 2024-04-06T20:20:43.000Z

I have been using the driver in existing deployments since Oct 2023 and the annotation was in spec.metadata.annotations. When did it move to spec.template.metadata.annotations? I spent several hours trying to debug a new deployment until I found this thread and moved the annotation to spec.template.metadata.annotations. Now it works!

I am sorry, I double-checked and the definition in my previous deployments was in spec.template.metadata.annotations.