Dynamic mounting fails
zchenyu opened this issue · 2 comments
zchenyu commented
Following these instructions here: https://cloud.google.com/kubernetes-engine/docs/how-to/persistent-volumes/cloud-storage-fuse-csi-driver#enable
I0420 21:27:13.940100 1 main.go:48] Running Google Cloud Storage FUSE CSI driver sidecar mounter version v0.1.14-gke.2
I0420 21:27:15.441183 1 sidecar_mounter.go:158] connecting to socket "/gcsfuse-tmp/.volumes/xxx/socket"
I0420 21:27:15.441349 1 fdchannel.go:48] get the underlying socket
I0420 21:27:15.441378 1 fdchannel.go:60] calling recvmsg...
I0420 21:27:15.441630 1 fdchannel.go:69] parsing SCM...
I0420 21:27:15.441655 1 fdchannel.go:76] parsing SCM_RIGHTS...
I0420 21:27:15.442091 1 main.go:122] waiting for SIGTERM signal...
I0420 21:27:15.442189 1 sidecar_mounter.go:71] start to mount bucket "_" for volume "xxx"
I0420 21:27:15.442579 1 sidecar_mounter.go:299] gcsfuse config file content:
logging:
file-path: /dev/fd/1
format: text
I0420 21:27:15.442690 1 sidecar_mounter.go:96] gcsfuse mounting with args [gcsfuse --app-name gke-gcs-fuse-csi --foreground --uid 0 --gid 0 --temp-dir /gcsfuse-buffer/.volumes/xxx/temp-dir --config-file /gcsfuse-tmp/.volumes/xxx/config.yaml --implicit-dirs _ /dev/fd/3]...
time="20/04/2024 09:27:15.462853" severity=INFO message="Start gcsfuse/1.4.1-gke.0 (Go version go1.21.5) for app \"gke-gcs-fuse-csi\" using mount point: /dev/fd/3\n"
time="20/04/2024 09:27:15.463290" severity=INFO message="GCSFuse mount command flags: {\"AppName\":\"gke-gcs-fuse-csi\",\"Foreground\":true,\"ConfigFile\":\"/gcsfuse-tmp/.volumes/xxx/config.yaml\",\"MountOptions\":{},\"DirMode\":493,\"FileMode\":420,\"Uid\":0,\"Gid\":0,\"ImplicitDirs\":true,\"OnlyDir\":\"\",\"RenameDirLimit\":0,\"CustomEndpoint\":null,\"BillingProject\":\"\",\"KeyFile\":\"\",\"TokenUrl\":\"\",\"ReuseTokenFromUrl\":true,\"EgressBandwidthLimitBytesPerSecond\":-1,\"OpRateLimitHz\":-1,\"SequentialReadSizeMb\":200,\"MaxRetrySleep\":30000000000,\"StatCacheCapacity\":4096,\"StatCacheTTL\":60000000000,\"TypeCacheTTL\":60000000000,\"HttpClientTimeout\":0,\"MaxRetryDuration\":-1000000000,\"RetryMultiplier\":2,\"LocalFileCache\":false,\"TempDir\":\"/gcsfuse-buffer/.volumes/xxx/temp-dir\",\"ClientProtocol\":\"http1\",\"MaxConnsPerHost\":100,\"MaxIdleConnsPerHost\":100,\"EnableNonexistentTypeCache\":false,\"StackdriverExportInterval\":0,\"OtelCollectorAddress\":\"\",\"LogFile\":\"\",\"LogFormat\":\"json\",\"ExperimentalEnableJsonRead\":false,\"DebugFuseErrors\":true,\"DebugFuse\":false,\"DebugFS\":false,\"DebugGCS\":false,\"DebugHTTP\":false,\"DebugInvariants\":false,\"DebugMutex\":false}"
time="20/04/2024 09:27:15.463499" severity=INFO message="GCSFuse mount config flags: {\"CreateEmptyFile\":false,\"Severity\":\"INFO\",\"Format\":\"text\",\"FilePath\":\"/dev/fd/1\",\"LogRotateConfig\":{\"MaxFileSizeMB\":512,\"BackupFileCount\":10,\"Compress\":true}}"
time="20/04/2024 09:27:15.463647" severity=INFO message="Creating Storage handle..."
time="20/04/2024 09:27:15.468178" severity=INFO message="Creating a mount at \"/dev/fd/3\"\n"
time="20/04/2024 09:27:15.468392" severity=INFO message="Creating a temporary directory at \"/gcsfuse-buffer/.volumes/xxx/temp-dir\"\n"
time="20/04/2024 09:27:15.468597" severity=INFO message="Creating a new server...\n"
time="20/04/2024 09:27:15.468696" severity=INFO message="Set up root directory for all accessible buckets"
time="20/04/2024 09:27:15.468829" severity=INFO message="Mounting file system \"gcsfuse\"..."
time="20/04/2024 09:27:15.469382" severity=INFO message="File system has been successfully mounted."
time="20/04/2024 09:27:15.894292" severity=ERROR message="LookUpInode: input/output error, Error in iterating through objects: storage: bucket doesn't exist"
time="20/04/2024 09:27:15.894405" severity=ERROR message="fuse: *fuseops.LookUpInodeOp error: input/output error"
time="20/04/2024 09:27:15.919100" severity=ERROR message="LookUpInode: input/output error, Error in iterating through objects: storage: bucket doesn't exist"
time="20/04/2024 09:27:15.919214" severity=ERROR message="fuse: *fuseops.LookUpInodeOp error: input/output error"
time="20/04/2024 09:27:16.186726" severity=ERROR message="LookUpInode: input/output error, Error in iterating through objects: storage: bucket doesn't exist"
time="20/04/2024 09:27:16.186833" severity=ERROR message="fuse: *fuseops.LookUpInodeOp error: input/output error"
time="20/04/2024 09:27:16.374238" severity=ERROR message="LookUpInode: input/output error, Error in iterating through objects: storage: bucket doesn't exist"
time="20/04/2024 09:27:16.374316" severity=ERROR message="fuse: *fuseops.LookUpInodeOp error: input/output error"
time="20/04/2024 09:27:16.599775" severity=ERROR message="LookUpInode: input/output error, Error in iterating through objects: storage: bucket doesn't exist"
time="20/04/2024 09:27:16.599912" severity=ERROR message="fuse: *fuseops.LookUpInodeOp error: input/output error"
time="20/04/2024 09:27:16.600955" severity=ERROR message="ReadDir: operation not supported, readAllEntries: ReadEntries: operation not supported"
time="20/04/2024 09:27:16.601045" severity=ERROR message="fuse: *fuseops.ReadDirOp error: operation not supported"
According to https://cloud.google.com/storage/docs/gcsfuse-mount#dynamic-mount , it seems we're supposed to exclude the bucket name from the gcsfuse command. It looks like the sidecar is trying to pass "_".
songjiaxun commented
Hi @zchenyu , did you directly run ls against the root mount point? I think that will give you errors.
If you have three buckets, my-bucket-1, my-bucket-2, and my-bucket-3, you won't see them as sub-folders in the root mount point. Instead, you will need to specify the bucket as part of the list operation. For example, ls /path/to/mount/point/my-bucket-1/. Or, you can cd to /path/to/mount/point/my-bucket-1/, then do ls.
Please let me know if it works.
zchenyu commented
Ohh thank you!! I had some init code that was implicitly listing that directory. The mount actually succeeded.