Use PHP to upload to gcs through gcs-fuse-csi-driver. It succeeds the first time but fails the second time.
880831ian opened this issue · 3 comments
880831ian commented
Hello, my current architecture is a deployment, which contains php-fpm and nginx, and I added a pv to it to use gcs-fuse-csi-driver. For details, you can view the following yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: active-dev
namespace: active
labels:
app: active-dev
spec:
replicas: 1
selector:
matchLabels:
app: active-dev
template:
metadata:
annotations:
gke-gcsfuse/volumes: "true"
gke-gcsfuse/cpu-limit: 20m
gke-gcsfuse/memory-limit: 20Mi
gke-gcsfuse/ephemeral-storage-limit: 1Gi
labels:
app: active-dev
spec:
serviceAccountName: k8s-gcs-sa
securityContext:
runAsUser: 0
initContainers:
- name: copy-application
image: asia-east1-docker.pkg.dev/xxxxx
imagePullPolicy: Always
command:
- /bin/bash
- "-c"
- |
cp -r . /mnt/application/
volumeMounts:
- name: application
mountPath: /mnt/application/
containers:
- name: php-fpm
image: asia-east1-docker.pkg.dev/xxxxx
imagePullPolicy: Always
command: ["/bin/bash", "-c", "php-fpm"]
ports:
- containerPort: 9000
name: http
protocol: TCP
securityContext:
privileged: true
volumeMounts:
- name: application
mountPath: /var/www/html
- name: php-fpm-log
mountPath: /var/log/php-fpm/
- name: php-fpm-config
mountPath: /usr/local/etc/php-fpm.d/www.conf
subPath: www.conf
- name: php-fpm-config
mountPath: /usr/local/etc/php/php.ini
subPath: php.ini
- name: storage
mountPath: /var/www/html/storage
- name: nginx
image: asia-east1-docker.pkg.dev/xxxxx
imagePullPolicy: Always
workingDir: /var/log/nginx
command: ["/bin/bash", "-c", "nginx -g 'daemon off;'"]
ports:
- containerPort: 80
- containerPort: 443
securityContext:
privileged: true
volumeMounts:
- name: application
mountPath: /var/www/html
- name: nginx-config
mountPath: /etc/nginx/nginx.conf
subPath: nginx.conf
- name: nginx-config
mountPath: /etc/nginx/conf.d/default.conf
subPath: default.conf
- name: storage
mountPath: /var/www/html/storage
volumes:
- name: application
emptyDir: {}
- name: php-fpm-log
emptyDir: {}
- name: php-fpm-config
configMap:
name: php-fpm-config
- name: nginx-config
configMap:
name: nginx-config
- name: storage
persistentVolumeClaim:
claimName: gcs-pvc
The settings for pv are as follows
apiVersion: v1
kind: PersistentVolume
metadata:
name: gcs-pv
spec:
accessModes:
- ReadWriteMany
capacity:
storage: 100Gi
storageClassName: standard-rwo
claimRef:
namespace: active
name: gcs-pvc
mountOptions:
- implicit-dirs
- dir-mode=0777
- file-mode=0777
- debug_fuse
- debug_gcs
csi:
driver: gcsfuse.csi.storage.gke.io
volumeHandle: xxx
The problem I am currently encountering is that I use php to upload images. The first time I can upload to gcs normally, but when I want to upload the second time, I will not be able to upload, and gke-gcsfuse-sidecar will also automatically close. I used s3fs before, and it was normal to use at that time. I don’t know if there are any restrictions.
I made sure I turned on GKE_METADATA on WI and node_pool, and the cluster version is also 1.27
Attached below are the successful and failed debug_fuse and debug_gcs logs
- successful
D1122 20:59:55.736790 fuse_debug: Op 0x00000068 connection.go:416] <- LookUpInode (parent 5, name "5UJx72ka1AZq.png", PID 0)
│ D1122 20:59:55.737601 gcs: Req 0x7: <- ListObjects("photefiles/5UJx72ka1AZq.png/")
│ D1122 20:59:55.737907 gcs: Req 0x8: <- StatObject("photefiles/5UJx72ka1AZq.png")
│ D1122 20:59:55.754091 gcs: Req 0x7: -> ListObjects("photefiles/5UJx72ka1AZq.png/") (16.51662ms): OK
│ D1122 20:59:55.947111 gcs: Req 0x8: -> StatObject("photefiles/5UJx72ka1AZq.png") (209.214353ms): OK
│ D1122 20:59:55.947786 fuse_debug: Op 0x00000068 connection.go:498] -> OK (inode 9)
│ D1122 20:59:55.948971 fuse_debug: Op 0x0000006a connection.go:416] <- LookUpInode (parent 1, name "photefiles", PID 0)
│ D1122 20:59:55.950761 fuse_debug: Op 0x0000006a connection.go:498] -> OK (inode 5)
│ D1122 20:59:55.951651 fuse_debug: Op 0x0000006c connection.go:416] <- LookUpInode (parent 5, name "5UJx72ka1AZq.png", PID 0)
│ D1122 20:59:55.952506 fuse_debug: Op 0x0000006c connection.go:498] -> OK (inode 9)
│ D1122 20:59:55.953134 fuse_debug: Op 0x0000006e connection.go:416] <- Unlink (parent 5, name "5UJx72ka1AZq.png", PID 0)
│ D1122 20:59:55.953749 gcs: Req 0x9: <- DeleteObject("photefiles/5UJx72ka1AZq.png")
│ D1122 20:59:55.981139 gcs: Req 0x9: -> DeleteObject("photefiles/5UJx72ka1AZq.png") (27.390122ms): OK
│ D1122 20:59:55.981175 fuse_debug: Op 0x0000006e connection.go:498] -> OK ()
│ D1122 20:59:56.046952 fuse_debug: Op 0x00000070 connection.go:416] <- SetInodeAttributes (inode 9, PID 0, mtime 2023-10-25 17:36:15.307 +0000 UTC)
│ D1122 20:59:56.047693 gcs: Req 0xa: <- UpdateObject("photefiles/5UJx72ka1AZq.png")
│ D1122 20:59:56.060874 gcs: Req 0xa: -> UpdateObject("photefiles/5UJx72ka1AZq.png") (13.180426ms): gcs.NotFoundError: storage: object doesn't exist
│ D1122 20:59:56.060895 gcs: Req 0xb: <- StatObject("photefiles/5UJx72ka1AZq.png")
│ D1122 20:59:56.075599 gcs: Req 0xb: -> StatObject("photefiles/5UJx72ka1AZq.png") (14.702217ms): gcs.NotFoundError: storage: object doesn't exist
│ D1122 20:59:56.075670 fuse_debug: Op 0x00000070 connection.go:498] -> OK ()
│ D1122 20:59:56.076233 fuse_debug: Op 0x00000072 connection.go:416] <- ForgetInode (inode 9, PID 0)
│ D1122 20:59:56.076359 fuse_debug: Op 0x00000072 connection.go:498] -> OK ()
│ D1122 20:59:56.079661 fuse_debug: Op 0x00000074 connection.go:416] <- LookUpInode (parent 1, name "photefiles", PID 0)
│ D1122 20:59:56.079792 fuse_debug: Op 0x00000074 connection.go:498] -> OK (inode 5)
│ D1122 20:59:56.147981 fuse_debug: Op 0x00000076 connection.go:416] <- LookUpInode (parent 5, name "X1RACUjmmRjE.png", PID 0)
│ D1122 20:59:56.148078 gcs: Req 0xc: <- ListObjects("photefiles/X1RACUjmmRjE.png/")
│ D1122 20:59:56.148274 gcs: Req 0xd: <- StatObject("photefiles/X1RACUjmmRjE.png")
│ D1122 20:59:56.247000 gcs: Req 0xc: -> ListObjects("photefiles/X1RACUjmmRjE.png/") (98.923571ms): OK
│ D1122 20:59:56.347548 gcs: Req 0xd: -> StatObject("photefiles/X1RACUjmmRjE.png") (199.280594ms): gcs.NotFoundError: storage: object doesn't exist
│ D1122 20:59:56.347654 fuse_debug: Op 0x00000076 connection.go:500] -> Error: "no such file or directory"
│ D1122 20:59:56.447070 fuse_debug: Op 0x00000078 connection.go:416] <- LookUpInode (parent 1, name "photefiles", PID 0)
│ D1122 20:59:56.448664 fuse_debug: Op 0x00000078 connection.go:498] -> OK (inode 5)
│ D1122 20:59:56.449697 fuse_debug: Op 0x0000007a connection.go:416] <- LookUpInode (parent 1, name "photefiles", PID 0)
│ D1122 20:59:56.450239 fuse_debug: Op 0x0000007a connection.go:498] -> OK (inode 5)
│ D1122 20:59:56.451638 fuse_debug: Op 0x0000007c connection.go:416] <- LookUpInode (parent 5, name "X1RACUjmmRjE.png", PID 0)
│ D1122 20:59:56.452748 gcs: Req 0xe: <- ListObjects("photefiles/X1RACUjmmRjE.png/")
│ D1122 20:59:56.148274 gcs: Req 0xd: <- StatObject("photefiles/X1RACUjmmRjE.png")
│ D1122 20:59:56.247000 gcs: Req 0xc: -> ListObjects("photefiles/X1RACUjmmRjE.png/") (98.923571ms): OK
│ D1122 20:59:56.347548 gcs: Req 0xd: -> StatObject("photefiles/X1RACUjmmRjE.png") (199.280594ms): gcs.NotFoundError: storage: object doesn't exist
│ D1122 20:59:56.347654 fuse_debug: Op 0x00000076 connection.go:500] -> Error: "no such file or directory"
│ D1122 20:59:56.447070 fuse_debug: Op 0x00000078 connection.go:416] <- LookUpInode (parent 1, name "photefiles", PID 0)
│ D1122 20:59:56.448664 fuse_debug: Op 0x00000078 connection.go:498] -> OK (inode 5)
│ D1122 20:59:56.449697 fuse_debug: Op 0x0000007a connection.go:416] <- LookUpInode (parent 1, name "photefiles", PID 0)
│ D1122 20:59:56.450239 fuse_debug: Op 0x0000007a connection.go:498] -> OK (inode 5)
│ D1122 20:59:56.451638 fuse_debug: Op 0x0000007c connection.go:416] <- LookUpInode (parent 5, name "X1RACUjmmRjE.png", PID 0)
│ D1122 20:59:56.452748 gcs: Req 0xe: <- ListObjects("photefiles/X1RACUjmmRjE.png/")
│ D1122 20:59:56.547027 gcs: Req 0xe: -> ListObjects("photefiles/X1RACUjmmRjE.png/") (94.286243ms): OK
│ D1122 20:59:56.547550 fuse_debug: Op 0x0000007c connection.go:500] -> Error: "no such file or directory"
│ D1122 20:59:56.647908 fuse_debug: Op 0x0000007e connection.go:416] <- CreateFile (parent 5, name "X1RACUjmmRjE.png", PID 0)
│ D1122 20:59:56.649350 gcs: Req 0xf: <- CreateObject("photefiles/X1RACUjmmRjE.png")
│ D1122 20:59:56.980882 gcs: Req 0xf: -> CreateObject("photefiles/X1RACUjmmRjE.png") (331.527469ms): OK
│ D1122 20:59:56.981489 fuse_debug: Op 0x0000007e connection.go:498] -> OK (inode 10)
│ D1122 20:59:56.981771 fuse_debug: Op 0x00000080 connection.go:416] <- GetXattr (inode 10, name "security.capability", PID 0, name security.capability)
│ D1122 20:59:56.981989 fuse_debug: Op 0x00000080 connection.go:500] -> Error: "function not implemented"
│ D1122 20:59:57.046965 fuse_debug: Op 0x00000082 connection.go:416] <- WriteFile (inode 10, PID 0, handle 6, offset 0, 377083 bytes)
│ D1122 20:59:57.047084 fuse_debug: Op 0x00000084 connection.go:416] <- SetInodeAttributes (inode 10, PID 0, mtime 2023-11-22 20:59:56.982887129 +0000 UTC)
│ D1122 20:59:57.047298 gcs: Req 0x10: <- Read("photefiles/X1RACUjmmRjE.png", <nil>)
│ D1122 20:59:57.064292 gcs: Req 0x10: -> Read("photefiles/X1RACUjmmRjE.png", <nil>) (17.008223ms): OK
│ D1122 20:59:57.065976 fuse_debug: Op 0x00000082 connection.go:498] -> OK ()
│ D1122 20:59:57.067015 fuse_debug: Op 0x00000084 connection.go:498] -> OK ()
│ D1122 20:59:57.067956 fuse_debug: Op 0x00000086 connection.go:416] <- FlushFile (inode 10, PID 0)
│ D1122 20:59:57.068776 gcs: Req 0x11: <- StatObject("photefiles/X1RACUjmmRjE.png")
│ D1122 20:59:57.084966 gcs: Req 0x11: -> StatObject("photefiles/X1RACUjmmRjE.png") (16.189164ms): OK
│ D1122 20:59:57.147036 gcs: Req 0x12: <- CreateObject("photefiles/X1RACUjmmRjE.png")
│ D1122 20:59:57.387487 gcs: Req 0x12: -> CreateObject("photefiles/X1RACUjmmRjE.png") (240.4477ms): OK
│ D1122 20:59:57.387734 fuse_debug: Op 0x00000086 connection.go:498] -> OK ()
│ D1122 20:59:57.387851 fuse_debug: Op 0x00000088 connection.go:416] <- ReleaseFileHandle (PID 0)
│ D1122 20:59:57.387895 fuse_debug: Op 0x00000088 connection.go:498] -> OK ()
│ D1122 20:59:57.446296 fuse_debug: Op 0x0000008a connection.go:416] <- LookUpInode (parent 1, name "photefiles", PID 0)
│ D1122 20:59:57.446600 fuse_debug: Op 0x0000008a connection.go:498] -> OK (inode 5)
│ D1122 20:59:57.448116 fuse_debug: Op 0x0000008c connection.go:416] <- LookUpInode (parent 5, name "X1RACUjmmRjE.png", PID 0)
│ D1122 20:59:57.448297 fuse_debug: Op 0x0000008c connection.go:498] -> OK (inode 10)
│ D1122 20:59:57.448947 fuse_debug: Op 0x0000008e connection.go:416] <- OpenFile (inode 10, PID 0)
│ D1122 20:59:57.449682 fuse_debug: Op 0x0000008e connection.go:498] -> OK ()
│ D1122 20:59:57.467401 fuse_debug: Op 0x00000090 connection.go:416] <- FlushFile (inode 10, PID 0)
│ D1122 20:59:57.467512 fuse_debug: Op 0x00000090 connection.go:498] -> OK ()
│ D1122 20:59:57.467790 fuse_debug: Op 0x00000092 connection.go:416] <- ReleaseFileHandle (PID 0)
│ D1122 20:59:57.467933 fuse_debug: Op 0x00000092 connection.go:498] -> OK ()
│ D1122 21:00:06.062589 fuse_debug: Op 0x00000094 connection.go:416] <- LookUpInode (parent 1, name "photefiles", PID 0)
│ D1122 21:00:06.064452 fuse_debug: Op 0x00000094 connection.go:498] -> OK (inode 5)
│ D1122 21:00:06.064543 fuse_debug: Op 0x00000096 connection.go:416] <- LookUpInode (parent 5, name "X1RACUjmmRjE.png", PID 0)
│ D1122 21:00:06.064632 fuse_debug: Op 0x00000096 connection.go:498] -> OK (inode 10)
│ D1122 21:00:06.064726 fuse_debug: Op 0x00000098 connection.go:416] <- LookUpInode (parent 1, name "photefiles", PID 0)
│ D1122 21:00:06.065219 fuse_debug: Op 0x00000098 connection.go:498] -> OK (inode 5)
│ D1122 21:00:06.065310 fuse_debug: Op 0x0000009a connection.go:416] <- LookUpInode (parent 5, name "X1RACUjmmRjE.png", PID 0)
│ D1122 21:00:06.066277 fuse_debug: Op 0x0000009a connection.go:498] -> OK (inode 10)
- failed
D1122 21:00:06.067244 fuse_debug: Op 0x0000009c connection.go:416] <- Unlink (parent 5, name "X1RACUjmmRjE.png", PID 0)
│ D1122 21:00:06.067287 gcs: Req 0x13: <- DeleteObject("photefiles/X1RACUjmmRjE.png")
│ D1122 21:00:06.096069 gcs: Req 0x13: -> DeleteObject("photefiles/X1RACUjmmRjE.png") (28.780515ms): OK
│ D1122 21:00:06.096186 fuse_debug: Op 0x0000009c connection.go:498] -> OK ()
│ D1122 21:00:06.096421 fuse_debug: Op 0x0000009e connection.go:416] <- SetInodeAttributes (inode 10, PID 0, mtime 2023-11-22 20:59:56.982887129 +0000 UTC)
│ D1122 21:00:06.096492 gcs: Req 0x14: <- UpdateObject("photefiles/X1RACUjmmRjE.png")
│ D1122 21:00:06.111028 gcs: Req 0x14: -> UpdateObject("photefiles/X1RACUjmmRjE.png") (14.535234ms): gcs.NotFoundError: storage: object doesn't exist
│ D1122 21:00:06.111118 gcs: Req 0x15: <- StatObject("photefiles/X1RACUjmmRjE.png")
│ D1122 21:00:06.129442 gcs: Req 0x15: -> StatObject("photefiles/X1RACUjmmRjE.png") (18.321316ms): gcs.NotFoundError: storage: object doesn't exist
│ D1122 21:00:06.130454 fuse_debug: Op 0x0000009e connection.go:498] -> OK ()
│ D1122 21:00:06.132780 fuse_debug: Op 0x000000a0 connection.go:416] <- LookUpInode (parent 1, name "photefiles", PID 0)
│ D1122 21:00:06.133412 fuse_debug: Op 0x000000a2 connection.go:416] <- ForgetInode (inode 10, PID 0)
│ D1122 21:00:06.133516 fuse_debug: Op 0x000000a2 connection.go:498] -> OK ()
│ D1122 21:00:06.134608 fuse_debug: Op 0x000000a0 connection.go:498] -> OK (inode 5)
│ D1122 21:00:06.136582 fuse_debug: Op 0x000000a4 connection.go:416] <- LookUpInode (parent 5, name "88SzdPTJaab5.png", PID 0)
│ D1122 21:00:06.136623 gcs: Req 0x16: <- ListObjects("photefiles/88SzdPTJaab5.png/")
│ D1122 21:00:06.136791 gcs: Req 0x17: <- StatObject("photefiles/88SzdPTJaab5.png")
│ D1122 21:00:06.247033 gcs: Req 0x16: -> ListObjects("photefiles/88SzdPTJaab5.png/") (110.399612ms): OK
│ D1122 21:00:06.348047 gcs: Req 0x17: -> StatObject("photefiles/88SzdPTJaab5.png") (211.251836ms): gcs.NotFoundError: storage: object doesn't exist
│ D1122 21:00:06.348279 fuse_debug: Op 0x000000a4 connection.go:500] -> Error: "no such file or directory"
│ D1122 21:00:06.349283 fuse_debug: Op 0x000000a6 connection.go:416] <- LookUpInode (parent 1, name "photefiles", PID 0)
│ D1122 21:00:06.447896 fuse_debug: Op 0x000000a6 connection.go:498] -> OK (inode 5)
│ D1122 21:00:06.448446 fuse_debug: Op 0x000000a8 connection.go:416] <- LookUpInode (parent 1, name "photefiles", PID 0)
│ D1122 21:00:06.548289 fuse_debug: Op 0x000000a8 connection.go:498] -> OK (inode 5)
│ D1122 21:00:06.548383 fuse_debug: Op 0x000000aa connection.go:416] <- LookUpInode (parent 5, name "88SzdPTJaab5.png", PID 0)
│ D1122 21:00:06.548411 gcs: Req 0x18: <- ListObjects("photefiles/88SzdPTJaab5.png/")
│ D1122 21:00:06.563771 gcs: Req 0x18: -> ListObjects("photefiles/88SzdPTJaab5.png/") (15.354685ms): OK
│ D1122 21:00:06.563907 fuse_debug: Op 0x000000aa connection.go:500] -> Error: "no such file or directory"
│ D1122 21:00:06.565011 fuse_debug: Op 0x000000ac connection.go:416] <- CreateFile (parent 5, name "88SzdPTJaab5.png", PID 0)
│ D1122 21:00:06.646952 gcs: Req 0x19: <- CreateObject("photefiles/88SzdPTJaab5.png")
│ D1122 21:00:06.880482 gcs: Req 0x19: -> CreateObject("photefiles/88SzdPTJaab5.png") (233.544459ms): OK
│ D1122 21:00:06.881030 fuse_debug: Op 0x000000ac connection.go:498] -> OK (inode 11)
│ D1122 21:00:06.882530 fuse_debug: Op 0x000000ae connection.go:416] <- WriteFile (inode 11, PID 0, handle 8, offset 0, 307106 bytes)
│ D1122 21:00:06.882970 fuse_debug: Op 0x000000b0 connection.go:416] <- SetInodeAttributes (inode 11, PID 0, mtime 2023-11-22 21:00:06.880761855 +0000 UTC)
│ D1122 21:00:06.946971 gcs: Req 0x1a: <- Read("photefiles/88SzdPTJaab5.png", <nil>)
│ D1122 21:00:06.968509 gcs: Req 0x1a: -> Read("photefiles/88SzdPTJaab5.png", <nil>) (21.564205ms): OK
│ D1122 21:00:07.047259 fuse_debug: Op 0x000000ae connection.go:498] -> OK ()
│ D1122 21:00:07.047421 fuse_debug: Op 0x000000b0 connection.go:498] -> OK ()
│ D1122 21:00:07.047560 fuse_debug: Op 0x000000b2 connection.go:416] <- FlushFile (inode 11, PID 0)
│ D1122 21:00:07.048541 gcs: Req 0x1b: <- StatObject("photefiles/88SzdPTJaab5.png")
│ D1122 21:00:07.066117 gcs: Req 0x1b: -> StatObject("photefiles/88SzdPTJaab5.png") (17.578269ms): OK
│ D1122 21:00:07.066238 gcs: Req 0x1c: <- CreateObject("photefiles/88SzdPTJaab5.png")
songjiaxun commented
Hi @880831ian ,
By looking at the yaml, I believe the issue is caused by insufficient memory allocation for the sidecar container. Could you please try increasing the memory to at least 50Mi? For example, change the Pod annotation to gke-gcsfuse/memory-limit: 50Mi.
Please let me know if it works for you. Thanks!
880831ian commented
@songjiaxun Wow, I tried to adjust gke-gcsfuse/memory-limit: 50Mi, and it succeeded for the second time, but it still got stuck the third time, so I changed gke-gcsfuse/memory-limit to the default value of 256Mi, and it has been tested so far. , everything is fine, Thak
songjiaxun commented
Glad that it works!