Expose parallel image pulls settings
Opened this issue · 8 comments
What I'd like:
To set the following via user data -
serialzeImagePulls:false
maxParallelImagePulls: 10
Are you referring to the kubelet options with these names? If so, are you using EKS and one of the bottlerocket aws-eks variants? If so, the eksctl tool does have a mechanism for adding custom configuration to the nodes it provisions. See https://eksctl.io/usage/customizing-the-kubelet/ for details.
If that doesn't meet your requirements, please do let us know which Bottlerocket variant you are running and (if these are the kubernetes options) how you configure and control your cluster, so we can better understand how we can help.
@larvacea that option doesn't work for Bottlerocket - changes to files in /etc, like kubelet's config, need to come through as settings so they can survive a node restart or settings change.
@ozahavi FWIW all aws-k8s-* AMIs default to serializeImagePulls: false, and maxParallelImagePulls is not set. Is your goal to limit the number of parallel image pulls?
@ozahavi FWIW all aws-k8s-* AMIs default to serializeImagePulls: false, and maxParallelImagePulls is not set. Is your goal to limit the number of parallel image pulls?
My goal is to pull in parallel. I am using bottlerocket-aws-k8s-1.27-x86_64-v1.19.5-64049ba8 and images are not pulled in parallel.
Pulling image "REDACTED.dkr.ecr.us-east-1.amazonaws.com/h2oai-dai-init-container:0.0.7"
2024-06-27T16:14:58+03:00
Pulling image "REDACTED.dkr.ecr.us-east-1.amazonaws.com/thirdparty-linkerd-proxyinit:v2.2.1"
2024-06-27T16:15:11+03:00
Pulling image "REDACTED.dkr.ecr.us-east-1.amazonaws.com/thirdparty-linkerd-proxy:v2.14.0"
2024-06-27T16:15:12+03:00
Pulling image "REDACTED.dkr.ecr.us-east-1.amazonaws.com/manager:0.7.0"
2024-06-27T16:15:13+03:00
+1
I wanted to clarify that the intention in this issue is to provide configuration of parallel pulls by exposing maxParallelImagePulls as described in #4176 so we'll track this feature under this issue.
Hi @ozahavi, serializeImagePulls is set to false by default; I confirmed that parallel pulls are indeed happening on a bottlerocket-aws-k8s-1.27-aarch64-v1.21.1-82691b51 node:
$ kubectl get --raw "/api/v1/nodes/<node-ip>.us-west-2.compute.internal/proxy/configz" | jq .kubeletconfig.serializeImagePulls
false
$ kubectl apply -f test-parallel-pull.yaml # podspec with ~2GB image (bottlerocket-core-kit)
$ kubectl apply -f test-parallel-pull-2.yaml # podspec with ~2GB image (bottlerocket-sdk)
I observed via kubectl get events that neither pull event spent any time waiting (note the duration and the "X including waiting" times are identical):
99s Normal Pulled pod/test-parallel-pull-core-kit Successfully pulled image "public.ecr.aws/bottlerocket/bottlerocket-core-kit:v2.4.0" in 58.261s (58.261s including waiting)
...
3s Normal Pulled pod/test-parallel-pull-sdk Successfully pulled image "public.ecr.aws/bottlerocket/bottlerocket-sdk:v0.44.0" in 2m51.041s (2m51.041s including waiting)
To be sure, I tested against a dev Bottlerocket AMI where I'm able to set serializeImagePulls to true, and observed the image pulls for one of my pods had to wait, an indication that the images were pulled in serial:
19s Normal Pulled pod/test-parallel-pull-core-kit Successfully pulled image "public.ecr.aws/bottlerocket/bottlerocket-core-kit:v2.4.0" in 1m14.109s (2m3.565s including waiting)
...
93s Normal Pulled pod/test-parallel-pull-sdk Successfully pulled image "public.ecr.aws/bottlerocket/bottlerocket-sdk:v0.44.0" in 52.597s (52.597s including waiting)
Paralell image pulling occurs at the pod level, not the container level: https://kubernetes.io/docs/concepts/containers/images/#serial-and-parallel-image-pulls Are your logs showing images for containers defined in the same pod?
The kubelet never pulls multiple images in parallel on behalf of one Pod. For example, if you have a Pod that has an init container and an application container, the image pulls for the two containers will not be parallelized. However, if you have two Pods that use different images, the kubelet pulls the images in parallel on behalf of the two different Pods, when parallel image pulls is enabled.
I'll also run a check with the variant you're using to be sure (bottlerocket-aws-k8s-1.27-x86_64-v1.19.5-64049ba8)
e: observed the same parallel pull behavior on the 1.19.5 node
ng occurs at the pod level, not the container level: https://kubernetes.io/docs/concepts/containers/images/#serial-and-parallel-image-pulls Are your logs showing images for containers defined in the same pod?
@ginglis13 Thanks for the clarification. I must have missed that multiple images cannot be pulled in parallel for one pod.
Knowing that, it seems the behavior is as expected.