Proposal: Working Group for Reference Types

Question

Proposal: Working Group for Reference Types

SteveLasker opened this issue 5 years ago · 34 comments

Working Group Proposal: Reference Types

Proposal created from OCI WG template.

Reference Types OCI Working Group - Governance Charter

This document describes the basic governance principles for the Reference Types Working Group (the “WG”).

The WG operates as an OCI Working Group under the Open Container Initiative (OCI) Charter, which describes the responsibilities of the OCI Technical Oversight Board (the "TOB”). The WG is established by the TOB as an OCI Working Group pursuant to the OCI Charter. Accordingly, the WG will operate in accordance with the OCI Charter and OCI's other policies and procedures, supplemented by the details below.

Purpose

As cloud native development continues to grow, there is increased interest in evolving registries to natively store, discover, and pull a graph of content associated with specific container images in a registry. Use cases for said associated artifacts include but are not limited to Software Bill of Materials (SBoM), security scan results, and signing. Having a native way to store and discover artifacts associated with other artifacts enables end-users to answer the question of: “What SBOMs or signatures are associated with this container image?”

Scope

Define and deliver the capability to store, discover, and pull a graph of artifacts associated with a specific artifacts to OCI distribution compliant registries. These set of capabilities has been commonly known as "reference types" or "references".
- Define supported use cases
- Document impact to existing user-facing tools and registries
- Define the method for creating, distributing, and discovering referenced objects
- Document user expectations for promoting an artifact between registries with its references
- Document onboaring process for registies and user-facing tools to adopt reference types
- Defined expectations for artifact reference lifecycle management
- Deliver a reference implementation of the reference types proposal

Out of Scope

Identified out of scope items will be listed here as WG progresses

Intended work product

Referrers API specification that provides the ability to discover references to existing container images. These include listing signatures, SBoMs, security scan results that refer to the digest of a manifest. The referrers API specification will sit within or along side the Distribution specification.
Identify, and document the pros and cons for versioning the existing manifests, compared with creating a new manifest to support reference types.

Proposed Owners

Lachlan Evenson (@lachie83)
Justin Cormack (@justincormack)
Michael Brown (@michaelb990)
Derek McGowan (@dmcgowan)
Jon Johnson (@jonjohnsonjr)

Related issues/PRs

Governance

Working Group:
- The TOB is establishing the WG as an OCI Working Group, pursuant to section 6(p) of the OCI Charter.
Owners:
- The WG proposal to the TOB will specify one or more initial "owners" of the WG.
- The current owners will be listed in the OCI Working Group documentation.
- The owners shall be responsible for:
  - scheduling regular meetings of the WG community;
  - facilitating open discussion among WG community participants;
  - coordinating and managing the development of the WG work product and outputs;
  - recording decisions that are reached by the WG community; and
  - keeping the TOB regularly informed about the status of the WG’s efforts, including when the WG has readied the work product and outputs for TOB approval.
Maintainers:
- If the WG owners request the TOB to approve a draft specification as a released OCI Specification, the request shall include a list of proposed "maintainers" of the OCI Specification.
- The current maintainers will be listed in the OCI Working Group documentation.
- The maintainers shall be responsible for continuing the work of overseeing updates, improvements and changes to a released OCI Specification on an ongoing basis.
Meetings:
- Meetings of the WG shall be open to the public.
- Participants in the meetings shall comply with the OCI Code of Conduct and all other policies of OCI and The Linux Foundation.
TOB Approval:
- The WG shall operate pursuant to the procedures set forth in section 6(p) of the OCI Charter, with regards to obtaining TOB approval for initial release of the work product and outputs as an OCI Specification or other OCI Project, and for subsequent maintenance activities thereafter.
Amendments:
- The owners of the WG may from time to time propose to the TOB (1) amendments to this WG Governance Document, and/or (2) changes to the composition of the owners or maintainers of the WG.
- As set forth in the OCI Charter, the TOB may, in its discretion by a two-thirds vote, approve or reject the requested amendments or changes.
- As set forth in the OCI Charter, the TOB may also disband the WG by a two-thirds vote.

Answer 1 · 2021-04-20T21:07:49.000Z

Is this group meant to supersede, replace or take over any of the existing proposals?

opencontainers/artifacts#37
opencontainers/artifacts#29
opencontainers/image-spec#828

Answer 2 · 2021-04-20T21:11:26.000Z

In accordance with last weeks working group discussion, this is a working group proposal to define the goals, objectives and, validations which should outline which of these we might pursue, or how we might pursue them in sequence to meet short-term and long term goals.

Answer 3 · 2021-04-20T21:18:47.000Z

Maybe this is something that needs to be cleared up in the working group proposal itself as it turns into an actual proposed charter change, but my understanding from reading it is that it's intended for the development and proposal of entirely new specifications (like the distribution example).

I think there's still some fundamental disagreement on whether or not reference types actually needs a new specification, or whether they can be accomplished via changes to the existing specifications. Proposing a WG here feels premature, especially before the WG model itself has been sorted out.

Answer 4 · 2021-04-20T21:22:33.000Z

Please see the scope section as we called out the specific concern:

Scope
Define enhancements to an existing spec or a new spec for how artifacts may support artifact reference types

Answer 5 · 2021-04-20T21:25:51.000Z

Right, the question is whether a WG is needed for minor additions to an existing spec.

Answer 6 · 2021-04-20T21:43:59.000Z

Agreed that it is something that needs to be clarified. I think with this working group, there needs to be a new specification or major revision which introduces such a manifest. For minor releases in a spec, there does not need to be working group. We still need to figure out in the distribution spec how new endpoints would be handled in minor releases.

The sponsors in this case should include the registry clients which will support this new feature. It is not intended to just be the companies which have engineers working on it, but the projects which will implement it.

What I am hoping the working group proposal method can do is unblock the decision process of what is being delivered since establishing the working group will be voted on. With that in mind, the scope should be explicit about what the expected result should be. For example, is this intending to deliver an artifacts-spec 1.0 or an image spec 2.0. If it has dependency on delivering a distribution-spec 1.1 (or 1.next) with a new endpoint, that should be called out. The distribution spec changes would not be released by this working group though, it could be part of the implementation process though.

Answer 7 · 2021-04-20T21:57:34.000Z

What I am hoping the working group proposal method can do is unblock the decision process of what is being delivered since establishing the working group will be voted on. With that in mind, the scope should be explicit about what the expected result should be. For example, is this intending to deliver an artifacts-spec 1.0 or an image spec 2.0. If it has dependency on delivering a distribution-spec 1.1 (or 1.next) with a new endpoint, that should be called out. The distribution spec changes would not be released by this working group though, it could be part of the implementation process though.

Well said!

To be extra clear - if minor, backwards compatible changes can be made to image-spec 1.0, I would expect that to still go through the standard image-spec proposal process defined and approved by the image-spec maintainers.

The image-spec group and maintainers could always decide or ask for a working group to help out with the implementation if they think it would help.

If we're not sure whether or not a major revision is needed, IMO the best thing to do would be to figure that out definitively then propose the next course of action, or even to try both paths simultaneously.

Answer 8 · 2021-08-16T16:06:10.000Z

With PR #99 merged, what are the next steps for the Reference Types working group?
@estesp, @vbatts,, @dmcgowan

Answer 9 · 2021-08-16T16:36:07.000Z

My guess as to process, from the charter:

iii. Approval of a new OCI Working Group requires a two-thirds vote of the TOB. The TOB will maintain a public list of all approved OCI Working Groups.

So this would need to be voted on by the TOB.

I think @dmcgowan's questions around exact scope and deliverables of this working group still need to be addressed in the proposal:

With that in mind, the scope should be explicit about what the expected result should be. For example, is this intending to deliver an artifacts-spec 1.0 or an image spec 2.0. If it has dependency on delivering a distribution-spec 1.1 (or 1.next) with a new endpoint, that should be called out. The distribution spec changes would not be released by this working group though, it could be part of the implementation process though.

Answer 10 · 2021-08-19T17:37:44.000Z

big agree with Derek.

To be clear, I want to see some form of reference types and the relevant APIs in distribution-spec. It seems to me this could be minimal, and therefore a point release (not a major version bump).

The extent to which there are new manifests needing to be defined, and possibly untangled from what is currently in image-spec, is related but slightly separate thread.

Answer 11 · 2021-08-20T19:22:16.000Z

Could we get some clarity from the owners of this proposal with respect to meaning of "reference types" reference and type are two very overloaded terms..

I think I see it being used in the context of referable objects (blobs) stored in a registry that have a known media type specified by oci or other such governing body. And in that context this means new base types in the same way that application/vnd.oci.image.manifest.v1+json and application/vnd.oci.image.config.v1+json are reference types.

If that is the context I'd like to see that clarified and perhaps also include in the scope providing suggestions for revisions to the image spec that identify base reference types, compatible with the existing image spec, for supporting the image spec reference types and new artifact reference type(s) ... Per discussions with @stevvooe one concern is that we do proper diligence in keeping our content-addressable storage (CAS) model intact (not immutable, no.. just be careful to ensure the extensions are compatible as we move forward).

Answer 12 · 2021-08-20T20:13:55.000Z

figuring out the scope/specifying this workgroup with such clarity would help to guide the changes/extensions being proposed... For example: one new field in a proposed artifiact manifest .. is artifactType and that is basically a "this".type string/media type where the manifest would now say what type it is.. Today we don't have a this reference in the object instead we normally specify the type of the blob(object) when/where we reference the object. Should this field be optional? is the name of the field obvious? is that a field we need on all base reference types? or is it only something we need on new artifact types. Not trying to get the answer to that question here.. Just proposing that such questions may prove important with respect to having a common CAS model and distribution api where the clients and registries know what to do with such fields..

Answer 13 · 2021-08-20T20:57:50.000Z

Thanks, @mikebrow.
Only through this discussion did I realize folks were reading "referenceTypes" to simple base types, such as string, descriptor, manifest, config...

This goes back to the definitions and terms conversation. Are we talking about the same thing.

For the purposes of the Working Group Proposal, we are discussing the ability to store reverse linkages between a new artifact being pushed to a registry, and existing content in a registry. That is the scope of the groups effort.

Answer 14 · 2021-08-20T21:26:29.000Z

thx for the clarification!

Answer 15 · 2021-08-24T13:08:31.000Z

Any updates here? Are you still planning to finish this up and submit it for approval?

Answer 16 · 2021-08-24T13:48:24.000Z

👍 Had a few chats recently, with jonjohnsonjr and stevelasker. I'm looking forward to the API pieces, and how this working group will tease out the definitions that ended up in image-spec in the times before distribution-spec was a reality.

Answer 17 · 2021-10-01T14:11:35.000Z

Can we get some kind of update here? This has been open since April and is effectively "holding a lock" on progress within the OCI.

Can it either be completed and submitted for a vote, or closed so someone else can finish it?

Answer 18 · 2021-10-01T14:42:58.000Z

update: It was suggested, discussed, and agreed at the OSS Summit Conference yesterday afternoon (near the end of the meeting on September, 30) that this working group would be brought to the tob for vote.

Answer 19 · 2021-10-01T17:19:55.000Z

Thanks @mikebrow! I'm looking for any sense of timeline for when this will be completed and brought for a vote. Days? Weeks? Months?

Answer 20 · 2021-10-19T18:07:32.000Z

Are there any updates here?

Answer 21 · 2021-10-19T18:10:47.000Z

Pinging @lachie83 who I think may have the next step(s) here based on a discussion we had last week? Apologies Lachie if I misunderstood.

Answer 22 · 2021-10-20T22:43:43.000Z

Hi all. We've updated the proposal based on feedback and would like to ask the TOB to review.

In addition, I would like to ask if @jonjohnsonjr would like to join the proposal as a registry operator to be both an owner and sponsor of this working group representing Google?

Answer 23 · 2021-10-21T00:47:20.000Z

This looks great to me!

Answer 24 · 2021-10-25T23:13:31.000Z

@lachie83 @jonjohnsonjr is willing, but is currently taking time off and probably won't be able to respond personally until early november

Answer 25 · 2021-10-27T19:42:10.000Z

the revised op-post looks good (i wish there were a way to get a permalink to the revision I just read 🤔 ).

One nit [however much it is in-the-weeds] is that as I'm reading through the existing distribution-spec, and with the anticipation for this kind of referrer API, I worry that the word "reference" will not be too overloaded, as it is already used in the distribution-spec.

Lastly, as @estesp has mentioned, I think this is ready for a [time-boxed] vote.

Answer 26 · 2021-11-09T18:28:18.000Z

Friendly ping.

Lastly, as @estesp has mentioned, I think this is ready for a [time-boxed] vote.

Has the time box opened? Is there a time box on opening the time box? 🤔

Answer 27 · 2021-11-09T20:26:30.000Z

Thanks @imjasonh; I was hoping for a verification that @jonjohnsonjr was going to be added formally to the proposal text based on the comment that he was willing but was still taking some time off. I'd like the TOB to vote on the specific list of collaborators/owners which is why I was ok with a delay until that was verified/updated.

If we can do that now, as soon as that's complete, let's set a 7-day window for any final commentary/questions and a vote on the day that comment period closes. Any other opinions here from @opencontainers/tob?

Answer 28 · 2021-11-10T23:13:53.000Z

I would like to see the voting started. Are we planning on making the working group proposals as files within the /proposals (or maybe under a subdirectory there)?

Answer 29 · 2021-11-11T00:59:39.000Z

In addition, I would like to ask if @jonjohnsonjr would like to join the proposal as a registry operator to be both an owner and sponsor of this working group representing Google?

Formal thumbs up 👍

Apologies @lachie83, I've had a lot to catch up on 😅

The sponsors in this case should include the registry clients which will support this new feature. It is not intended to just be the companies which have engineers working on it, but the projects which will implement it.

Was this feedback from @dmcgowan addressed? I'm happy to be an owner on this, but would like to see sponsorship from more projects that would be affected (both registries and clients). IMO, some representation from at least Quay and Harbor would be excellent.

Answer 30 · 2021-11-18T18:52:52.000Z

18 Nov OCI call discussion: Let's update owners with Jon's name. Copy the proposal text into a markdown file; commit that as a new file in the proposals/ directory of this repo and open a PR. That PR will have the TOB vote, target end date of 1 December (due to US holidays next week).

Answer 31 · 2021-11-18T19:00:49.000Z

Just a point of clarity,
I've added @jonjohnsonjr above, as the template for what will be copied into the PR.
As we often have people represent their personal views compared to company positions, I just wanted to clarify if we're saying Google is a sponsor as well?

Answer 32 · 2022-08-09T18:50:31.000Z

Isn't this issue resolved, as this WG is in full swing?

Answer 33 · 2022-08-09T19:00:00.000Z