kubernetes/enhancements

VolumeSource: OCI Artifact and/or Image

sallyom opened this issue · 121 comments

sallyom commented

Enhancement Description

sallyom commented

I've had informal discussions about this - there's enough interest IMO to open a KEP & I will present this issue at upcoming sig-node, sig-storage mtgs with a KEP draft

/sig node
/sig storage

  • ❤️1
xing-yang commented

Can you use the Volume Populator? It allows you to create a PVC from an extenal data source. https://github.com/kubernetes/enhancements/tree/master/keps/sig-storage/1495-volume-populators

  • 👀1
saschagrunert commented

Happy to support here from a SIG node perspective.

cc @kubernetes/sig-node-proposals

mikebrow commented

+1 happy to help from the SIG-Node/CRI and OCI image/distribution spec perspectives..

SergeyKanzhelev commented

/label lead-opted-in
/milestone v1.31

as discussed at SIG Node meeting this week, we will try and see if this can make it to 1.31

SergeyKanzhelev commented

/stage alpha

rhuss commented

For reference, in KServe a workaround for directly accessing files within an OCI image is currently implemented and available via a sidecar approach ("modelcar") by leveraging root FS system access via the /proc filesystem when shareProcessNamespace: true is set on the Pod. You can find details in the KServe documentation and in the Design Document. It actually implements the desired behavior with current means, but of course is more or less just a workaround for the OCI volume type (as discussed here and raised already a long time ago in kubernetes/kubernetes#831).

So KServe would be more than happy to leverage such a volume type, and we are happy to support any efforts in this direction.

  • 👍3
sreeram-venkitesh commented

/stage alpha

sreeram-venkitesh commented

Hello @sallyom 👋, v1.31 Enhancements team here.

Just checking in as we approach enhancements freeze on 02:00 UTC Friday 14th June 2024 / 19:00 PDT Thursday 13th June 2024.

This enhancement is targeting for stage alpha for v1.31 (correct me, if otherwise)

Here's where this enhancement currently stands:

  • KEP readme using the latest template has been merged into the k/enhancements repo.
  • KEP status is marked as implementable for latest-milestone: v1.31. KEPs targeting stable will need to be marked as implemented after code PRs are merged and the feature gates are removed.
  • KEP readme has up-to-date graduation criteria
  • KEP has a production readiness review that has been completed and merged into k/enhancements. (For more information on the PRR process, check here). If your production readiness review is not completed yet, please make sure to fill the production readiness questionnaire in your KEP by the PRR Freeze deadline (6th June) so that the PRR team has enough time to review your KEP before the enhancements freeze.

For this KEP, most of the above items are taken care of in #4642. We'd need to do the following:

  • Update the status from provisional to implementable in the kep.yaml file here
  • Create a prod-readiness yaml file as shown here.
  • Update the graduation criteria in the KEP readme file.
  • Make sure that the PRR questionnaire is filled.

The status of this enhancement is marked as At risk for enhancements freeze. Once the above tasks are done, I can mark it as tracked.

If you anticipate missing enhancements freeze, you can file an exception request in advance. Let me know if you have any questions! Thank you!

sreeram-venkitesh commented

@sallyom Pinging once again as a slight reminder that we're approaching the enhancements freeze deadline on 14th June, this Friday!

dipesh-rawat commented

Hi @sallyom @SergeyKanzhelev 👋, 1.31 Enhancements team here,

Just a quick friendly reminder as we approach the enhancements freeze in few hours, at 02:00 UTC Friday 14th June 2024 / 19:00 PDT Thursday 13th June 2024.

The current status of this enhancement is marked as at risk for enhancement freeze. There are a few requirements mentioned in the comment #4639 (comment) that are addressed as part of PR #4642 which still needs to be merged.

If you anticipate missing enhancements freeze, you can file an exception request in advance. Thank you!

dipesh-rawat commented

Hello @sallyom @SergeyKanzhelev 👋, 1.31 Enhancements team here.

Unfortunately, this enhancement did not meet requirements for enhancements freeze.

If you still wish to progress this enhancement in v1.31, please file an exception request as soon as possible, within three days. If you have any questions, you can reach out in the #release-enhancements channel on Slack and we'll be happy to help. Thanks!

  • 😕1
sreeram-venkitesh commented

/milestone clear

bitoku commented

/assign

sreeram-venkitesh commented

Marked as tracked for enhancements freeze! Thanks everyone!

Princesso commented

Hello @bitoku, @sallyom 👋, 1.31 Docs Lead here.
Does this enhancement work planned for 1.31 require any new docs or modifications to existing docs?
If so, please follow the steps here to open a PR against dev-1.31 branch in the k/website repo. This PR can be just a placeholder at this time and must be created before Thursday June 27, 2024 18:00 PDT.
Also, take a look at Documenting for a release to get yourself familiarised with the docs requirement for the release.
Thank you!

saschagrunert commented

@Princesso yes, I created a documentation placeholder PR in kubernetes/website#46946

saschagrunert commented

/assign

hailkomputer commented

Hi @saschagrunert , @sallyom

👋 from the v1.31 Communications Team! We'd love for you to opt in to write a feature blog about your enhancement!
Some reasons why you might want to write a blog for this feature include (but are not limited to) if this introduces breaking changes, is important to our users, or has been in progress for a long time and is graduating.

To opt in, let us know and open a Feature Blog placeholder PR against the website repository by 3rd July, 2024. For more information about writing a blog see the blog contribution guidelines.

Note: In your placeholder PR, use XX characters for the blog date in the front matter and file name. We will work with you on updating the PR with the publication date once we have a final number of feature blogs for this release.

saschagrunert commented

@hailkomputer thank you for the hint, I added a placeholder PR in kubernetes/website#46960

  • 👍1
utam0k commented

@saschagrunert Do you know where the containerd side is implemented?
containerd has a special behavior when pulling images following their registry configuration. I would like to know if this behavior is the same when retrieving from OCI Artifact, so I would like to see the implementation.
https://github.com/containerd/containerd/blob/main/docs/cri/config.md#registry-configuration

saschagrunert commented

@saschagrunert Do you know where the containerd side is implemented? containerd has a special behavior when pulling images following their registry configuration. I would like to know if this behavior is the same when retrieving from OCI Artifact, so I would like to see the implementation. https://github.com/containerd/containerd/blob/main/docs/cri/config.md#registry-configuration

I would assume that we have no implementation right now for containerd. @mikebrow may provide more details on that.

  • 👍1
sreeram-venkitesh commented

Hey again @saschagrunert @sallyom 👋 v1.31 Enhancements team here,

Just checking in as we approach code freeze at 02:00 UTC Wednesday 24th July 2024 / 19:00 PDT Tuesday 23rd July 2024.

Here's where this enhancement currently stands:

  • All PRs to the Kubernetes repo that are related to your enhancement are linked in the above issue description (for tracking purposes).
  • All PR/s are ready to be merged (they have approved and lgtm labels applied) by the code freeze deadline. This includes tests.

For this enhancement, it looks like the following PRs are open and need to be merged before code freeze (and we need to update the Issue description to include all the related PRs of this KEP):

Out of tree:

If you anticipate missing code freeze, you can file an exception request in advance.

I'm marking this KEP as At risk for code freeze for now since all the associated PRs haven't been approved/lgtm'd. But everything looks good here, thanks a lot for keeping the issue description up to date with all the PRs! I can update the KEP to tracked as soon as the code PRs are approved. Also, please let me know if there are other PRs in k/k we should be tracking for this KEP. As always, we are here to help if any questions come up. Thanks!

  • 👍1
sreeram-venkitesh commented

/milestone v1.31

sreeram-venkitesh commented

Hi again @sallyom @saschagrunert! We are one week away from code freeze and I wanted to ping you here as a friendly reminder for making sure the code PRs are merged in time before the deadline. Also, please let me know if there are any more PRs that needs to be tracked other than the ones linked in #4639 (comment). Thanks!

sreeram-venkitesh commented

@sallyom @saschagrunert All the PRs mentioned in #4639 (comment) are merged. Are we waiting for cri-o/cri-o#8408 and kubernetes/kubernetes#126220 to be merged as part of this KEP too? Please let me know!

saschagrunert commented

@sreeram-venkitesh we're now actively working on the docs, the implementation is done and everything which is still open is nice to have. Out of tree PR's are not a requirement for this KEP.

sreeram-venkitesh commented

Thanks @saschagrunert! Marking this KEP as tracked for code freeze 🎉

  • ❤️2
sreeram-venkitesh commented

The PR kubernetes/kubernetes#126220 which adds tests is still open. Please make sure to get it merged by the test freeze deadline (01:00 UTC Wednesday 31st July 2024 / 19:00 PDT Tuesday 30th July 2024).

@saschagrunert @sallyom

utam0k commented

@saschagrunert FYI: This is the issue for containerd to support this KEP.
containerd/containerd#10496

saschagrunert commented

@utam0k thank you for tracking the enhancement into containerd. Let me know if you need anything as support!

  • ❤️2
asviel commented

Hi everyone. Are there plans to support subPath or (better) projected volumes for OCI VolumeSource? I have multiple (10+) ML models and would like to be able to mount them in one place:

apiVersion: v1
kind: Pod
metadata:
  name: image-volume
spec:
  containers:
    - name: shell
      command: ["sleep", "infinity"]
      image: debian
      volumeMounts:
        - name: models
          mountPath: /models
  volumes:
    - name: models
      projected:
        sources:
          - image:
              reference: quay.io/crio/artifact-1:v1
              pullPolicy: IfNotPresent
          - image:
              reference: quay.io/crio/artifact-2:v1
              pullPolicy: IfNotPresent
saschagrunert commented

@asviel maybe in some later graduation of the feature. Thank you for the input here!

  • ❤️1
deepforu47 commented

Hello Everyone,
Anyone able to test image volume using Minikube with pod man and ImageVolume featureGate is enabled.

I have tried and getting below error-
Error: failed to generate container "cc992779b80f62ff8a8db99d005ab1caa6b282e18f2a6eb162cc5b2d882eb283" spec: failed to generate spec: failed to mkdir "": mkdir : no such file or directory

pod.yaml

apiVersion: v1
kind: Pod
metadata:
  name: image-volume
spec:
  containers:
  - name: shell
    command: ["sleep", "infinity"]
    image: busybox
    volumeMounts:
    - name: volume
      mountPath: /volume
  volumes:
  - name: volume
    image:
      reference: quay.io/crio/alpine:3.9
      pullPolicy: Always
saschagrunert commented

@deepforu47 please open an issue in https://github.com/kubernetes/kubernetes. I assume you're using containerd? Right now only CRI-O main supports it.

deepforu47 commented

Sure @saschagrunert will open new issue.
On your comment yes I was using containers, then tried with CRI-O as well. I saw documentation that it will work with CRI-O > 1.31(which I can't see http://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable:/cri-o:/).

New Error -

Warning Failed 2s kubelet Error: mount.HostPath is empty

mikebrow commented

@deepforu47 @saschagrunert FYI for tracking wip pr for containerd v2 release candidate: containerd/containerd#10579

  • ❤️2
wzshiming commented

Hi everyone, I’m proposing to add a new MediaType to the OCI image spec that will only hold raw data files, for optimization of large files in the image, please take a look.
opencontainers/image-spec#1197

sudo-bmitch commented

Over on the OCI side, we've been seeing a lot of confusion on this new feature. People are getting the impression that OCI Artifacts are supported, while the non-goals exclude everything except an OCI Image from being supported without directly saying that OCI Artifacts are not supported. Can this be clarified in the KEP and blog posts, removing all references to OCI Artifacts or placing a large disclosure at the top, to avoid confusing users?

For reference, the guidance for creating an OCI Artifact is linked in the KEP already: https://github.com/opencontainers/image-spec/blob/main/artifacts-guidance.md. A key requirement listed there is that the config media type is not application/vnd.oci.image.config.v1+json, and it allows layer blobs with any arbitrary binary data, not limiting it to the tar layers.

pacoxu commented

kubernetes/kubernetes#108405 (comment)

  • Should oci volume pulling block other pod image pulls if the kubelet uses the default serialize-image-pulls? Currently, we shared the same image puller. A big model image pulling may block other new pod image pulling(startup) in the worst case.
  • And should we add this to the beta or GA criteria to change the image pull behavior or not count OCI volume as part of pod image pulling? Or is this expected behavior?
  • 👍1
saschagrunert commented
  • Should oci volume pulling block other pod image pulls if the kubelet uses the default serialize-image-pulls? Currently, we shared the same image puller. A big model image pulling may block other new pod image pulling(startup) in the worst case.
  • And should we add this to the beta or GA criteria to change the image pull behavior or not count OCI volume as part of pod image pulling? Or is this expected behavior?

Thank you for the input here! I personally think we should make as less exceptions in behavior as possible for OCI image volumes in comparison to regular images.

tjons commented

Hi, enhancements lead here - I inadvertently added this to the 1.32 tracking board 😀. Please readd it if you wish to progress this enhancement in 1.32.

/remove-label lead-opted-in

saschagrunert commented

@tjons there is no graduation planned for this feature in v1.32.

  • 👍1
haircommander commented

/milestone v1.32
/label lead-opted-in

tjons commented

@saschagrunert @haircommander is there still no graduation planned for this feature in 1.32? If so, is the plan to iterate on alpha? and if not, why is this tracked for 1.32 :)?

jenshu commented

Hello @saschagrunert @sallyom 👋, Enhancements team here.

Just checking in as we approach enhancements freeze on 02:00 UTC Friday 11th October 2024 / 19:00 PDT Thursday 10th October 2024.

This enhancement is targeting stage alpha for v1.32 (correct me, if otherwise)

Here's where this enhancement currently stands:

  • KEP readme using the latest template has been merged into the k/enhancements repo.
  • KEP status is marked as implementable for latest-milestone: v1.32.
  • KEP readme has up-to-date graduation criteria
  • KEP has a production readiness review that has been completed and merged into k/enhancements. (For more information on the PRR process, check here). If your production readiness review is not completed yet, please make sure to fill the production readiness questionnaire in your KEP by the PRR Freeze deadline on Thursday 3rd October 2024 so that the PRR team has enough time to review your KEP.

For this KEP, we would just need to update the following:

The status of this enhancement is marked as at risk for enhancement freeze. Please keep the issue description up-to-date with appropriate stages as well.

If you anticipate missing enhancements freeze, you can file an exception request in advance. Thank you!

sohankunkerkar commented

@saschagrunert @sallyom, I'm part of the SIG-NODE KEPs wrangler team for this release. It looks like the plan for this release is to push this feature to Beta. Could one of you open a PR to move it forward?

haircommander commented

@saschagrunert is out this week, but I'm going to open a PR in his stead to get it on PRR's purview, then he'll probably correct all the incorrect stuff I've written next week 🙂

haircommander commented

update: I've opened #4897

haircommander commented

/stage beta

haircommander commented

/stage alpha
/remove-milestone v1.32
/remove-label lead-opted-in

I believe we won't make stage progress in 1.32. We can still push for subpath support but I don't think we need to track that here.

jenshu commented

1.32 Enhancements team here. Per the above comments, I've updated the 1.32 status to deferred

tjons commented

/milestone clear

tarilabs commented

Hi 👋
from the "model-spec-discussion" CNCF slack channel,
from the last meeting 2024-12-05,
we wanted to raise to this great KEP-4639 efforts,
one suggestion to be evaluated that maybe could also help see motivations for OCI Artifact to be indeed in the longer-term a goal for this KEP-4639 and the Container Runtimes.

One of the great advantage of the OCI Artifact from the OCI spec, is the possibility to define media_type (and/or artifact_type) with a semantic meaning.

This would potentially allow a user to mount only a set of determined layers instead of the whole OCI Image (as it is the case in practice today with alpha), ie something ~similar to:

  volumes:
    - name: volume
      image:
        reference: quay.io/crio/artifact:v1
        layer_mediaType_filter_IN: ["application/x-mlmodel"] # mount only layer(s) from Artifact which is the ML model, not Data layers, etc.

or like:

  volumes:
    - name: volume
      image:
        reference: quay.io/crio/artifact:v1
        layer_mediaType_filter_IN: ["application/x", "application/y"] # mount only layer(s) marked for X and Y, and for example ignoring layers of type Z if the Artifact provides them

please notice these examples (and naming this layer_mediaType_filter_IN) are in no means representative of the actual Yaml/Spec being suggested, but only to substantiate in a more concrete term an idea which was found meaningful from many within the group.

As one can imagine, if the single OCI Artifact is comprehensive of many assets duly separated in their own semantic layers, but only some layers are functional to runtime scope, this would be a great advantage for the definition of the Deployment workload. wdyt?

I hope this comment is helpful to the wide CNCF community and of interest in the scope of the continued work for this KEP-4639 🚀
cc @bmicklea , @bergwolf , @amisevsk , @chlins , @tarilabs (this), @gorkem , @sabre1041

  • 👍9
  • 🚀6
sudo-bmitch commented

we wanted to raise to this great KEP-4639 efforts,
one suggestion to be evaluated that maybe could also help see motivations for OCI Artifact to be indeed in the longer-term a goal for this KEP-4639 and the Container Runtimes.

One of the great advantage of the OCI Artifact from the OCI spec, is the possibility to define media_type (and/or artifact_type) with a semantic meaning.

When mounting other media types, what are the filesystem settings for the mounted layer? Is something extracted from a tar, requiring all Artifacts to be packaged as a tar? Or is the Artifact layer mounted as a file? How is the filename set, owner, permissions, and other filesystem parameters? From my understanding, the KEP authors were not ready to answer those questions and intentionally marked OCI Artifacts as out of scope for now, leaving individual runtimes free to experiment with their own implementations.

amisevsk commented

From my perspective, tarballs are a natural starting point for packaging OCI artifact layers, and these layers are generally going to be compatible with OCI image layer types (e.g. in testing the existing alpha-level feature, I'm able to mount artifacts by just changing the mediaType on layers and generating a simple image config).

Considering that the current state of the proposal is limited to OCI images, it's not a large leap to extend support for any OCI artifact whose media types are compatible with OCI image layers. The mediatype filtering field would then be an assertion from the user that these mediaTypes can be processed like image layers.

Alternatively, we could consider making this explicit in some way -- runtimes understand certain media types (tar, tar+gzip, etc.); does it make sense to enable defining artifact media types as synonyms for "known" mediatypes?

  • 👍3
gaius-qi commented

Alternatively, we could consider making this explicit in some way -- runtimes understand certain media types (tar, tar+gzip, etc.); does it make sense to enable defining artifact media types as synonyms for "known" mediatypes?

I think it's a good idea for container runtimes to understand certain media types (tar, tar+gzip, etc.). As User Stories describes in KEP, we can mount large language model weights, binary, etc. If only image media types are used, it will be confusing to understand what is stored. However, if there are no restrictions on artifact media types, various custom media types need to be handled, which will bring a lot of work.

  • 👍1
dmesser commented

I concur. I wonder, though, if there is a discussion already in the runtime space, how to adopt OCI artifacts or if it needs to be kicked off?

sabre1041 commented

I concur. I wonder, though, if there is a discussion already in the runtime space, how to adopt OCI artifacts or if it needs to be kicked off?

I have not seen any recent conversations, but it is a topic that would be good to raise and get additional mindshare

  • 👍5
saschagrunert commented

As you folks already outlined: runtimes are free to do anything with the feature. I don't think that Kubernetes itself should maintain the list of allowed and supported media types, that would be a better fit for the OCI. But said, runtimes like CRI-O are free to support any format which makes sense.

  • 👍2
  • ❤️3
iholder101 commented

Hey!
Just wanted to reach out and express my support for this feature.

In Kubevirt, we have the concept of container-disks, which are ephemeral disk images that can be added to VMs. We currently hack our way around to expose this into the main VM container, then mount it into the guest. This KEP would eliminate the need for our hacks and will help us remove a lot of code that would be officially supported.

A POC is already taking place here: kubevirt/kubevirt#13656.

I'll gladly help push this KEP forward, feel free to ping me for any help I can provide!

  • 👍1
  • 🚀1
Barakmor1 commented

Hey, I'm curious to know why subPath is disallowed for image volumes. Is this expected to change?

saschagrunert commented

Hey, I'm curious to know why subPath is disallowed for image volumes. Is this expected to change?

Hey, yes we aim to change that in the beta graduation and mainly did it for simplicity reasons. 🙃

  • 👍3
saschagrunert commented

Graduation depends on containerd/containerd#10579

  • 👍1
haircommander commented

/label lead-opted-in
/milestone v1.33

fykaa commented

Hello @sallyom @saschagrunert 👋, v1.33 Enhancements team here.

Just checking in as we approach enhancements freeze on 02:00 UTC Friday 14th February 2025 / 19:00 PDT Thursday 13th February 2025.

This enhancement is targeting stage beta for v1.33 (correct me, if otherwise)
/stage beta

Here's where this enhancement currently stands:

  • KEP readme using the latest template has been merged into the k/enhancements repo.
  • KEP status is marked as implementable for latest-milestone: v1.33.
  • KEP readme has up-to-date graduation criteria
  • KEP has a production readiness review that has been completed and merged into k/enhancements. (For more information on the PRR process, check here). If your production readiness review is not completed yet, please make sure to fill the production readiness questionnaire in your KEP by the PRR Freeze deadline on Thursday 6th February 2025 so that the PRR team has enough time to review your KEP.

For this KEP, we would just need to update the following:

  • Please update the Graduation criteria in the KEP's README
  • Please update the approver details for PRR
  • Please update the status as implementable for latest-milestone: v1.33.

The status of this enhancement is marked as At risk for enhancements freeze. Please keep the issue description up-to-date with appropriate stages as well.

If you anticipate missing enhancements freeze, you can file an exception request in advance. Thank you!

saschagrunert commented

#4897 will graduate the featuer to beta

  • 🎉4
  • 🚀3
dipesh-rawat commented

Hi @sallyom @saschagrunert 👋, 1.33 Enhancements team here,

Just a quick friendly reminder as we approach the enhancements freeze later this week, at 02:00 UTC Friday 14th February 2025 / 19:00 PDT Thursday 13th February 2025.

The current status of this enhancement is marked as At risk for enhancement freeze. There are a few requirements mentioned in the comment #4639 (comment) that still need to be completed.

If you anticipate missing enhancements freeze, you can file an exception request in advance. Thank you!

hacktivist123 commented

Hello @sallyom @saschagrunert 👋, v1.33 Docs Shadow here.

Does this enhancement work planned for v1.33 require any new docs or modification to existing docs?
If so, please follow the steps here to open a PR against dev-1.33 branch in the k/website repo. This PR can be just a placeholder at this time and must be created before Thursday 27th February 2025 18:00 PDT.

Also, take a look at Documenting for a release to get yourself familiarize with the docs requirement for the release.
Thank you!

dipesh-rawat commented

Hello @sallyom @saschagrunert 👋, 1.33 Enhancements team here.

Now that PR #4897 has been merged, all the KEP requirements are in place and merged into k/enhancements.

Before the enhancement freeze, it would be appreciated if following nit could be addressed:

  • Update issue description to mention
    • Beta release target (x.y): 1.33

Aside from the minor nit mentioned above, this enhancement is all good for the upcoming enhancements freeze. 🚀

The status of this enhancement is now marked as tracked for enhancement freeze. Please keep the issue description up-to-date with appropriate stages as well. Thank you!
(cc: @fykaa)

  • 👍1
saschagrunert commented
  • Beta release target (x.y): 1.33

Thank you, I updated the issue to match the new target release!

  • 👍1
Barakmor1 commented

Hey,

As mentioned in previous comments, this feature could be very useful for the KubeVirt operator. During some experimental use , we identified a few issues that need to be addressed. I'm not sure if these are intentional or documented, so I'd appreciate any help in reviewing or resolving them:

[Link] The ImageLocality scheduling plugin doesn’t account for ImageVolume images when scoring and prioritizing nodes with required pod images.

[Link] When using the restricted profile with PSA, pods with ImageVolumes are not allowed, and we get the following error: pods "virt-launcher-vmi-kernel-boot-94rcf-bklpk" is forbidden: violates PodSecurity "restricted:latest": restricted volume types (volume "kernel-boot" uses restricted volume type "unknown")

  • 👍1
gbartolini commented

This feature will be highly beneficial for our CloudNativePG operator (CNCF Sandbox project) to dynamically mount Postgres extensions as independent and self-contained images, ensuring they remain immutable. See this discussion: https://www.postgresql.org/message-id/flat/E7C7BFFB-8857-48D4-A71F-88B359FADCFD%40justatheory.com

rayandas commented

Hi @saschagrunert @sallyom 👋 v1.33 Docs Lead here.

Gentle reminder! The deadline to raise a placeholder Docs PR is Thursday 27th February 2025 18:00 PDT.
If this enhancement work requires any new docs or modification to existing docs, please create a placeholder PR (it can be a draft PR for now) before the deadline.

Thanks!

saschagrunert commented

Here is the docs placeholder PR: kubernetes/website#49936

  • 👍1
rytswd commented

Hi @sallyom @saschagrunert 👋 -- this is Ryota (@rytswd) from the 1.33 Communications Team!

For the 1.33 release, we are currently in the process of collecting and curating a list of potential feature blogs, and we'd love for you to consider writing one for your enhancement!

As you may be aware, feature blogs are a great way to communicate to users about features which fall into (but not limited to) the following categories:

  • This introduces some breaking change(s)
  • This has significant impacts and/or implications to users
  • ...Or this is a long-awaited feature, which would go a long way to cover the journey more in detail 🎉

To opt in to write a feature blog, could you please let us know and open a "Feature Blog placeholder PR" (which can be only a skeleton at first) against the website repository by Wednesday, 5th March, 2025? For more information about writing a blog, please find the blog contribution guidelines 📚

Tip

Some timeline to keep in mind:

  • 02:00 UTC Wednesday, 5th March, 2025: Feature blog PR freeze
  • Monday, 7th April, 2025: Feature blogs ready for review
  • You can find more in the release document

Note

In your placeholder PR, use XX characters for the blog date in the front matter and file name. We will work with you on updating the PR with the publication date once we have a final number of feature blogs for this release.

fykaa commented

Hey again @sallyom @saschagrunert 👋, v1.33 Enhancements team here,

Just checking in as we approach Code Freeze at 02:00 UTC Friday 21st March 2025 / 19:00 PDT Thursday 20th March 2025.

Here's where this enhancement currently stands:

  • All PRs to the Kubernetes repo that are related to your enhancement are linked in the above issue description (for tracking purposes).
  • All PRs are ready to be merged (they have approved and lgtm labels applied) by the code freeze deadline. This includes tests.

For this enhancement, it looks like the following PRs need to be merged before code freeze (and we need to update the Issue description to include all the related PRs of this KEP):

If you anticipate missing code freeze, you can file an exception request in advance.

Also, please let me know if there are other PRs in k/k we should be tracking for this KEP.

The status of this enhancement is marked as At risk for code freeze.

As always, we are here to help if any questions come up. Thanks!

gbartolini commented

This is a blog article that I just published that shows how CloudNativePG - now a Sandbox project - will benefit from this feature to load immutable container images for Postgres extensions dynamically: "The Immutable Future of PostgreSQL Extensions in Kubernetes with CloudNativePG".

  • ❤️1
rytswd commented

Hi @sallyom @saschagrunert 👋, 1.33 Communications Team here again!

This is a gentle reminder for the feature blog deadline mentioned above, which is 02:00 UTC Wednesday, 5th March, 2025. To opt in, please let us know and open a Feature Blog placeholder PR against k/website by the deadline. If you have any questions, please feel free to reach out to us!

Tip

Some timeline to keep in mind:

  • 02:00 UTC Wednesday, 5th March, 2025: Feature blog PR freeze
  • Monday, 7th April, 2025: Feature blogs ready for review
  • You can find more in the release document

Note

In your placeholder PR, use XX characters for the blog date in the front matter and file name. We will work with you on updating the PR with the publication date once we have a final number of feature blogs for this release.

saschagrunert commented

I created the feature blog: kubernetes/website#49984

  • ❤️2
rytswd commented

Thanks @saschagrunert! I have associated the placeholder PR with this KEP under the Enhancement project board 👍

  • ❤️1
kkkkun commented

/cc

chenditc commented

I didn't see a discussion link for this KEP, so I will try to raise my concern in this thread:

As each layer is a tar with compression, initialize the volume from image will require downloading the tar and untar the files. If the image is big (GB+), the time it takes might exceeds the timeout. We see such issue when we use very large container image (10G+).

Would we try to address this issue in our roadmap? Like setting flexible timeout according to image size? Or provide configurable timeout value?

In the scenario of large language model, the tar and gzip doesn't reduce much data transfer time. And the tar + zip increase the download and unpack time, is it possible to also set "nocompression" flag for these volume image?

saschagrunert commented

@chenditc we pull the images in the sync pod context which should not timeout from the kubelet perspective. Means the overall pull should behave in the same way as for container images. Reasonably large images can be pulled by runtimes either without a timeout or a "timeout when no progress is made over a specific period of time" (a CRI-O feature).

Do you have any specific reproducer in mind?

sudo-bmitch commented

Layers packaged in tar, without gzip compression, are also already supported by OCI and runtimes that I've seen. So I'd be curious what implementation does not support this already.

mikebrow commented

pushing container images to the container registry with compression off (example: --disable-compression) is a client tooling feature that may or may not be available containers/buildah#3904

+1 to sasha's comment -- sync pod context means before the pod is run, kubelet requests the images to be pulled locally normally this is done with unpack,... containerd's 1.7/2.0 config.toml entry for the default timeout is image_pull_progress_timeout = '5m0s'

chenditc commented

Thanks for sharing valuable insight!

We were using kubernetes version 1.30.4 + containerd 1.6.26 + pulling image from Azure acr. The image is build using docker with LLM model weight from hugging face: https://huggingface.co/microsoft/Phi-3.5-vision-instruct/tree/main.

When a pod pulling the container image (20G+), if it timeout or failed due to various reason, the danling image layer won't get garbage collect right away, leaving some orphant tar file. The retry will worse the scenario, as there were less ephemeral disk to use. The size of the ephemeral disk is just enough to do a "happy day" download and untar, so it's very fragile to any failure.

@saschagrunert I don't have a stable reproducer yet, as this issue seems happen more frequently in environment with weak network or lower ephemeral disk.

@sudo-bmitch @mikebrow Thanks for sharing the option to disable compression, I will try disable the image compression and see if that helps.

One additional concern is the quota management for ephemeral disk. As the volume image is pull using similar mechanism as container image, it will use some ephemeral disk, but it doesn't have a clear isolation boundary like pvc, if volume image pull encounter error, the ephemeral disk pressure might impact all pods on that node.

fykaa commented

Hello @sallyom @saschagrunert 👋,

The v1.33 Enhancements team here again!

With all the implementation (code-related) PRs merged as per the issue description:

I noticed that kubernetes/kubernetes#130681 is currently marked as optional. Could you confirm whether it’s required for this release, or if we can consider the enhancement complete? If not, we’ll go ahead and mark it as Tracked for Code Freeze for v1.33.

Additionally, are there any other PRs in k/k that we should be tracking for this KEP to ensure an accurate status update?

Thanks! 🚀

saschagrunert commented

@fykaa code for this enhancement is complete for now, I don't think that the optional e2e test will land because we have no runtime support yet.

  • 👍2
  • 🚀1
Barakmor1 commented

Hi,

Currently in the containerStatuses field of the pod status, the imageID field includes the Digest suffix for each container, like this:

imageID: ...@sha256:f8110bad01b7e1a08b961613f38d104f4f8e05e66cea12fbda17e2ab206fc857

This is quite useful for detecting when an image has changed, especially for those using the latest tag. Is there any plan to include similar information for the imageVolume in the pod's status?

  • 👍2
dipesh-rawat commented

@saschagrunert Thanks, for the update and confirming that all required changes are merged (here). We can mark this as tracked for code freeze. Also, please let us know if anything changes before the freeze or if there are any other PRs in k/k we should track for this KEP to keep the status accurate.

This enhancement is now marked as tracked for code freeze for the v1.33 Code Freeze!

  • ❤️2
  • 🚀2
jenshu commented

Hi @saschagrunert 👋, 1.34 Enhancements Lead here.

I am closing the v1.33 milestone now.

If you'd like to work on this enhancement in v1.34, please have the SIG lead opt-in by adding the lead-opted-in label, which ensures it gets added to the tracking board. Also, please set the milestone to v1.34 using /milestone v1.34.

Thanks!

/remove-label lead-opted-in
/remove-label tracked/yes
/milestone clear

  • 👍1
saschagrunert commented

@jenshu thanks for reaching out! There is no graduation planned for v1.34. 👍

iholder101 commented

Hey @saschagrunert!

@jenshu thanks for reaching out! There is no graduation planned for v1.34. 👍

Thanks for the heads-up.
Do you mind sharing why graduation is not planned, or what are the significant blockers / remaining issues? Thank you!

In addition, regarding the comment above,

Currently in the containerStatuses field of the pod status, the imageID field includes the Digest suffix for each container, like this:

imageID: ...@sha256:f8110bad01b7e1a08b961613f38d104f4f8e05e66cea12fbda17e2ab206fc857

This is quite useful for detecting when an image has changed, especially for those using the latest tag.

Do you think this is something we can plan for GA?
Would it help id I'd open an issue, or create a KEP update PR to add this to GA requirements? WDYT?

Thanks!

theory commented

Where would be the appropriate venue to request that this feature support adding image volumes without restarting a Pod? The CloudNativePG project is looking at using it to deploy Postgres extensions via OCI images, and most don't require a cluster restart, just the insertion of the files. Would be nice not to have to restart the database.

bitoku commented

@iholder101 @Barakmor1 @saschagrunert

To implement the imageID in podStatus, technically we'd need to change api and cri-api
ImageSpec.Image shouldn't contain the canonical image reference like the one wanted.

we may want to open a new KEP.

  • 👍2
haircommander commented

/label lead-opted-in
/milestone v.1.34

k8s-ci-robot commented

@haircommander: The provided milestone is not valid for this repository. Milestones in this repository: [v1.25, v1.27, v1.28, v1.29, v1.30, v1.31, v1.32, v1.33, v1.34, v1.35]

Use /milestone clear to clear the milestone.

In response to this:

/label lead-opted-in
/milestone v.1.34

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

haircommander commented

/milestone v1.34

kannon92 commented

It is difficult to follow what work is needed for this KEP for 1.34.