Image index media type and annotation

Question

Image index media type and annotation

ok-ryoko opened this issue a year ago · 13 comments

For tighter integration with some container registries such as ghcr.io, I want to annotate multi-architecture image indexes. I understand that only media of type application/vnd.oci.image.index.v1+json support annotations. However, I’m unable to create an image index of this type using either Buildah or Podman. Moreover, there doesn’t seem to be a way to annotate image indexes—only images referenced by an index.

Steps to reproduce the issue:

$ buildah manifest create manticore && buildah manifest inspect manticore

Describe the results you received:

f25e51ee5f0a3be38ae63b954aa4a2367c20e514a9f66a007975ee1db8e9a217
{
    "schemaVersion": 2,
    "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
    "manifests": null
}

Describe the results you expected:

f25e51ee5f0a3be38ae63b954aa4a2367c20e514a9f66a007975ee1db8e9a217
{
    "schemaVersion": 2,
    "mediaType": "application/vnd.oci.image.index.v1+json",
    "manifests": null
}

… because, as I understand it, conformance with OCI specifications is a high priority to Buildah, and Buildah defaults to using the OCI format in other operations, such as committing and pushing images.

Output of buildah version:

Version:         1.31.3
Go Version:      go1.19.12
Image Spec:      1.1.0-rc.3
Runtime Spec:    1.1.0-rc.3
CNI Spec:        1.0.0
libcni Version:
image Version:   5.26.2
Git Commit:
Built:           Fri Aug 25 08:51:43 2023
OS/Arch:         linux/amd64
BuildPlatform:   linux/amd64

In cmd/buildah/manifest.go, the manifest type is currently fixed to manifest.DockerV2ListMediaType.

Could manifest create take a --format option (the same way that push and commit do) and respect the BUILDAH_FORMAT environment variable?

Could manifest annotate annotate the image index directly if (1) the media type is appropriate and (2) no image manifest digest is provided?

Answer 1 · 2023-09-18T15:18:24.000Z

I'll check this thanks.

Answer 2 · 2023-09-18T15:43:17.000Z

Way back when, when an image index could be valid without a mediaType field, an index with no mediaType and manifests provided basically nothing to go on for logic that was trying to guess its MIME type, specifically whether or not a chunk of JSON represented a list or a runnable image. That created problems elsewhere, so we defaulted to create lists in the Docker format, in which mediaType was a required field.

In practice, a list gets converted along with images when they're pushed somewhere with --format, or we choose a format when we update a local copy based on whether or not it contains information that's unique to one format or the other.

Not having a way to add annotations to the index as a whole via the command line is a different chunk of code.

Answer 3 · 2023-09-18T17:36:26.000Z

@nalind Thank you for providing context. I have indeed noticed that inspecting some manifests reveals their mediaType to be application/vnd.oci.image.index.v1+json after pushing.

If we were hypothetically able to annotate an image index, would we thus expect to start with the Docker type and switch to the OCI type automatically upon annotation (assuming there are no incompatible manifests)?

Answer 4 · 2023-09-18T17:54:06.000Z

When a list is being edited locally, when it's time to save the list, I'd expect the current logic to select the format that preserves any format-specific information when it goes to save the list in local storage. Things like URL or Annotation fields in the OCI format, or a Features field in the Docker format.

Unless it's changed since I was last in there, I don't think buildah itself cares about any correspondence between the mediaType of the list and the mediaType of images in the list -- it's not going to rewrite local images, which would change their IDs. Registries tend to care about that, so the image library tries to make everything match up on the fly, if they don't already, while a list is being pushed.

Answer 5 · 2023-09-21T18:16:27.000Z

When a list is being edited locally, when it's time to save the list, I'd expect the current logic to select the format that preserves any format-specific information when it goes to save the list in local storage. Things like URL or Annotation fields in the OCI format, or a Features field in the Docker format.

Based on my reading of the source code, I’d expect this behavior as well.

Unless it's changed since I was last in there, I don't think buildah itself cares about any correspondence between the mediaType of the list and the mediaType of images in the list -- it's not going to rewrite local images, which would change their IDs. Registries tend to care about that, so the image library tries to make everything match up on the fly, if they don't already, while a list is being pushed.

I confirmed that this is the case. When running buildah manifest push, Buildah sets the image index type according to the --format option. Buildah then defers the push operation to containers/common/libimage, which normalizes the manifest type for single images according to the type of the image index.

Thank you again for the context.

Answer 6 · 2023-10-17T06:47:21.000Z

@ok-ryoko Is your issue resolved ? I'm closing this issue since this looks resolved please free to re-open if it is not so.

Answer 7 · 2023-10-25T16:19:53.000Z

@flouthoc, thank you for following up.

The discussion helped me understand why application/vnd.docker.distribution.manifest.list.v2+json is the default image index type and why this doesn’t pose problems when an OCI-type image index is desired at push time.

However, there’s still no way to annotate image indexes using Buildah.

We currently have the following command that we can use to “[a]dd and update information about an image to a manifest list or image index”:

buildah manifest annotate [options...] listNameOrIndexName imageManifestDigest

To support image index annotation, we could (for example) make the imageManifestDigest argument optional such that, when imageManifestDigest is omitted, the annotation(s) apply to the image index itself.

As image index annotations are an OCIv1-specific feature (source), this operation would require the conversion of the image index type to application/vnd.oci.image.index.v1+json and would need to know how to handle conflicts appropriately.

This feature enhancement is something I’d be willing to design and contribute. Before writing any code, however, I’d like to get confirmation that this feature enhancement aligns with the project’s current vision and priorities.

And perhaps modifying an existing command isn’t the best way to go about introducing this functionality?

p.s. I’m not able to re-open the issue.

Answer 8 · 2023-11-25T00:09:35.000Z

A friendly reminder that this issue had no activity for 30 days.

Answer 9 · 2023-11-25T20:03:33.000Z

@nalind @flouthoc @umohnani8 Thoughts?

Answer 10 · 2024-01-25T18:34:02.000Z

@flouthoc There seems to be a clear benefit to have annotation on the image index, are there any updates on whether this is going to be added any time soon? Thanks!

Answer 11 · 2024-01-25T20:17:02.000Z

I'll visit this again and update back here thanks.

Answer 12 · 2024-01-25T21:59:18.000Z

I'm already in this area, might as well assign it to me.

Answer 13 · 2024-01-26T05:34:41.000Z

Thank you @nalind , did it 😄