Failed to create storage handle using createStorageHandle

[vparmeland]

Describe the issue
Sometimes my pods remain in error on creation
toto-xxxxxx-d6z6b 3/4 CreateContainerError 1 (51m ago) 54m
I have to kill the pod for it to start normally

System & Version:

• Platform [GKE 1.29]
• Version [FUSE CSI driver sidecar 1.4.3]

Steps to reproduce the behavior with following information:

 ---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: redacted-pv
spec:
  accessModes:
  - ReadOnlyMany
  capacity:
    storage: 500Gi
  storageClassName: toto-storage-class
  claimRef:
    namespace: {{ .Values.namespace }}
    name: redacted-pv
  mountOptions:
    - implicit-dirs
    - uid=1001
    - gid=3003
  csi:
    driver: gcsfuse.csi.storage.gke.io
    volumeHandle: "{{ .Values.global.gcpProject }}-toto-files"
    volumeAttributes:
      gcsfuseLoggingSeverity: warning  
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: redacted-pv
  namespace: {{ .Values.namespace }}
spec:
  accessModes:
  - ReadOnlyMany
  resources:
    requests:
      storage: 500Gi
  volumeName: redacted-pv
  storageClassName: toto-storage-class
---

12:52:22.342681       1 main.go:48] Running Google Cloud Storage FUSE CSI driver sidecar mounter version v1.4.3-gke.8 
12:52:23.845214       1 sidecar_mounter_config.go:101] connecting to socket "/gcsfuse-tmp/.volumes/redacted-pv/socket"
12:52:23.848660       1 fdchannel.go:48] get the underlying socket 
12:52:23.848701       1 fdchannel.go:60] calling recvmsg... 
12:52:23.912438       1 fdchannel.go:69] parsing SCM...     
12:52:23.912467       1 fdchannel.go:76] parsing SCM_RIGHTS...  
12:52:23.912821       1 sidecar_mounter_config.go:269] gcsfuse config file content: map[cache-dir: logging:map[file-path:/dev/fd/1 format:json severity:warning]] 
12:52:23.928456       1 sidecar_mounter.go:51] start to mount bucket "gcp-bucket-toto-file" for volume "redacted-pv"  
12:52:23.929020       1 sidecar_mounter.go:70] gcsfuse mounting with args [--app-name gke-gcs-fuse-csi --foreground --uid 1001 --gid 3003 --temp-dir /gcsfuse-buffer/.volumes/redacted-pv/temp-dir --config-file /gcsfuse-tmp/.volumes/redacted-pv/config.yaml --implici 
12:52:23.929503       1 main.go:73] waiting for SIGTERM signal... 
12:52:24.056520       1 sidecar_mounter.go:105] gcsfuse for bucket "gcp-bucket-toto-file", volume "redacted-pv" started with process id 14  
{"timestamp":{"seconds":1726577548,"nanos":656330481},"severity":"ERROR","message":"Error while mounting gcsfuse: Failed to create storage handle using createStorageHandle: go storage client creation failed: while creating http endpoint: while fetching tokenSource: DefaultTokenSource: google: could not find def │
Failed to create storage handle using createStorageHandle: go storage client creation failed: while creating http endpoint: while fetching tokenSource: DefaultTokenSource: google: could not find default credentials. See https://cloud.google.com/docs/authentication/external/set-up-adc for more information  
12:52:28.664260       1 logger.go:60] gcsfuse exited with error: exit status 1  
14:12:55.327789       1 main.go:110] received SIGTERM signal, waiting for all the gcsfuse processes exit... 
14:12:55.327925       1 main.go:118] exiting sidecar mounter... 
 Stream closed EOF for namespacea/toto-xxxxx-d6z6b (gke-gcsfuse-sidecar)

Thanks

[raj-prince]

Hi @vparmeland,

The failure is related to auth configuration mechanism in GKE for gcsfuse-csi-driver. GCSFuse is behaving as expected, it requires application default credential to mount GCS bucket which is not there is your case, hence the failure.

https://github.com/GoogleCloudPlatform/gcs-fuse-csi-driver is the better place to create the issue.

Overall guide to access GCS bucket with gcsfuse CSI driver - https://cloud.google.com/kubernetes-engine/docs/how-to/persistent-volumes/cloud-storage-fuse-csi-driver

Specifically GKE authentication using Workload Identity Federation: https://cloud.google.com/kubernetes-engine/docs/how-to/persistent-volumes/cloud-storage-fuse-csi-driver#authentication

[ankitaluthra1]

@vparmeland Was this issue resolved? please make sure cluster is setup correctly with workload identity, these requirements should be met for gcsfuse csi driver to work in addition to setting up bucket access as mentioned in previous comments.