Allow the gcsfuse sidecar container to use unlimited resource on GKE Standard clusters.
songjiaxun opened this issue · 6 comments
This is a feature request: to allow the gcsfuse sidecar container to use unlimited resource on GKE Standard clusters.
Some context.. GCS fuse is not usable for me because I work in a dynamic environment where sometimes nodes have a lot of CPU, memory and ephemeral storage and on those nodes I want to use maximum performance of GCS fuse without any limits such that my GPU workloads do not become bottlenecked or get evicted when there wasn't a need to evice them.
Right now my main container is hitting the ephemeral storage limit:
Warning Evicted 60s kubelet Pod ephemeral local storage usage exceeds the total limit of containers 100Gi.
However, I also need to run on nodes that only have 100 Gi
Expected behavior
GCS fuse should not set limits by default unless the user explicitly sets a limit
Steps to reproduce:
- Create a pod with following definition
apiVersion: v1
kind: Pod
metadata:
name: gcs-fuse-csi-example-ephemeral
namespace: default
annotations:
gke-gcsfuse/volumes: "true"
spec:
terminationGracePeriodSeconds: 60
containers:
- image: ubuntu
name: ubuntu
command: ["sleep"]
args: ["infinity"]
volumeMounts:
- name: gcs-fuse-csi-ephemeral
mountPath: /data
readOnly: true
serviceAccountName: modeller
volumes:
- name: gcs-fuse-csi-ephemeral
csi:
driver: gcsfuse.csi.storage.gke.io
readOnly: true
volumeAttributes:
bucketName: mah-bucket
mountOptions: "implicit-dirs"
- Write 10GB of files in the main ubuntu container (note I'm NOT writing to
/data) :
k exec -ti gcs-fuse-csi-example-ephemeral -c ubuntu -- bash
time sh -c 'dd if=/dev/zero iflag=count_bytes count=10G bs=1M of=large; sync'
- Observe the eviction due to exceeding ephemeral limit of the sidecar container:
Warning Evicted 20s (x2 over 60s) kubelet Pod ephemeral local storage usage exceeds the total limit of containers 5Gi.
Why is this bad for UX?
Even when I don't use the mounted path of GCS I will get hit by the ephemeral limit that GCS fuse introduces in the side car.
In the recent release, the gcsfuse sidecar container can consume unlimited resources on Standard clusters. if you pass "0" to the pod annotations, the sidecar container will have no resource limit set, which makes the pod burstable. For example:
apiVersion: v1
kind: Pod
metadata:
name: gcs-fuse-csi-example-ephemeral
namespace: default
annotations:
gke-gcsfuse/volumes: "true"
gke-gcsfuse/cpu-limit: "0"
gke-gcsfuse/memory-limit: "0"
gke-gcsfuse/ephemeral-storage-limit: "0"
The new CSI version is included in the following GKE versions or later versions:
- 1.24.17-gke.2155000
- 1.25.14-gke.1474000
- 1.26.9-gke.1507000
- 1.27.6-gke.1506000
- 1.28.2-gke.1157000
That's amazing. I will try it out soon!
Is this going to wok in Autopilot? I've quickly tried and it seems not.
No it won't work on autopilot because autopilot requires you to set a limit.