RFC: Buildpack Distribution Specification [WIP]

Question

RFC: Buildpack Distribution Specification [WIP]

sclevine opened this issue 6 years ago · 19 comments

NOTE: Work in progress

Motivation

This proposal enables both decentralized, manual distribution of buildpacks from Docker registries as well as automatic distribution of buildpacks from a centralized buildpack registry. It allows individual buildpacks to be distributed via a Docker registry, and it makes dynamic assembly of builders on a Docker registry efficient. It provides a developer-friendly interface that abstracts away complex buildpack configurations.

What it is

This RFC proposes an official way to distribute buildpacks that conform to the CNB buildpack v3 specification. Changes would consist of a new Buildpack Distribution specification, modifications the lifecycle builder, and modifications to the pack CLI.

It affects all personas that interact with buildpacks.

How it Works

CNB Package Format

A CNB package may exist as an OCI image on an image registry, an OCI image in a Docker daemon, or a .cnb file.

A .cnb file is an uncompressed tar archive containing an OCI image. Its file name should end in .cnb.

Each FS layer blob in the image must contain a single file or single populated directory in one of the following formats:

Buildpack

/cnb/buildpack/<buildpack ID>/<buildpack version>/

Default Buildpack Version Symlink

/cnb/buildpack/<buildpack ID>/default -> <buildpack version>/

Note that this symlink replaces the existing latest symlink.

Order

/cnb/order/<order ID>.toml

In comparsion to the order.toml format, the <order ID>.toml format that replaces it accepts an order ID in the id field of any given entry in [[groups.buildpacks]], provided that the version and optional fields are not specified.

To determine which buildpacks run, the detector processes the default.toml file (or <order ID>.toml file if manually specified) the same as the current order.toml format, with each order ID expanded as such:

Where:

O and P are objects containing order IDs
A through H are objects containing buildpack IDs
L and M are group (row) labels

Given:

$O = \begin{bmatrix} A_L, & B_L \\ C, & D \end{bmatrix}$

$P = \begin{bmatrix} E_M, & F_M \\ G, & H \end{bmatrix}$

We propose:

$\begin{bmatrix} E, & O, & F \end{bmatrix} = \begin{bmatrix} E_L, & A_L, & B_L, & F_L \\ E, & C, & D, & F \\ \end{bmatrix}$

$\begin{bmatrix} O, & P \end{bmatrix} = \begin{bmatrix} A_{LM}, & B_{LM}, & E_{LM}, & F_{LM} \\ A_{L}, & B_{L}, & G_{L}, & H_{L} \\ C_{M}, & D_{M}, & E_{M}, & F_{M} \\ C, & D, & G, & H \\ \end{bmatrix}$

(@hone: this is my interpretation of the logic you suggested at summit. Let me know if I misinterpreted.)

Default Order Symlink

/cnb/order/default.toml -> <order ID>.toml

CNB Package Metadata

All supported stacks must be provided in the OCI image metadata.

Label: io.buildpacks.cnb.metadata
JSON:
{
  "stacks": {
    "id": "io.buildpacks.stacks.bionic",
    "mixins": ["mysql"]
   }
}

For a CNB package to be valid, each buildpack.toml must have all listed stacks. Each build ID should only be present once, and the mixins list should enumerate all the required mixins for that stack for all included buildpacks.

User Interface

App Developer

pack build should accept a list of buildpacks and group IDs via the --buildpack flag. Additionally, it should accept a list of labels via the --label flag. Labels filter such that all specified labels must be present on a group for it to be considered for detection.

Buildpack Developer

pack create-cnb will package a selection of:

all buildpacks from selected CNB packages
all <order ID>.toml files from selected CNB packages
additional buildpacks
additional <order ID>.toml files that reference any of the above

into a .cnb file, OCI image in a registry, or OCI image in a Docker daemon. [TBD: format for cnb.toml file that specifies the location of these artifacts, default symlinks, default.toml symlinks, and stacks]

Instead of builder.toml, pack create-builder will generate a builder image from a CNB package and stack ID.

Unanswered Questions

Should order definitions be versioned?

Drawbacks

Adding multi-group order definitions together is complex.

Alternatives

No other RFCs are proposed.

Answer 1 · 2019-04-12T19:23:51.000Z

This RFC proposes a new way to distribute buildpacks...

I'd say "specified way" or "official way" as there is nothing defined to date. We're all doing random stuff.

Answer 2 · 2019-04-12T19:26:06.000Z

A .cnb file is an uncompressed tar archive containing an OCI image.

Should this be clarified that it should be in a docker load-compatible format?

Answer 3 · 2019-04-12T19:28:37.000Z

Default Buildpack Version Symlink

Why wouldn't every single release of a buildpack have this link pointing to the single version of the buildpack contained with in it?

Answer 4 · 2019-04-12T19:32:13.000Z

Can you please add a concrete example of the /cnb/order/<order ID>.toml?

Answer 5 · 2019-04-12T19:33:23.000Z

Default Order Symlink

Again like above, why wouldn't a packaged buildpack group release always point to the single <order ID>.toml file that it's going to package?

Answer 6 · 2019-04-13T01:36:36.000Z

Should this be clarified that it should be in a docker load-compatible format?

Unfortunately, no: moby/moby#25779

Can you please add a concrete example of the /cnb/order/.toml?

Will do. (It's the same as the existing order.toml, but it allows referencing other order ID files in place of buildpacks.)

Why wouldn't every single release of a buildpack have this link pointing to the single version of the buildpack contained with in it?

A CNB package that has a single version of a single buildpack should have a layer for the default version symlink if the consumer of the CNB package should be able to select that buildpack by ID without specifying a version. (The "consumer" may be a order ID definition or app developer.)

Again like above, why wouldn't a packaged buildpack group release always point to the single <order ID>.toml file that it's going to package?

A CNB package may have many of these <order ID>.toml files for organizational purposes. For example, you can use an <order ID>.toml to add APM agent buildpacks to every group without repeating the APM agent buildpack ID. A CNB package that is missing this symlink requires the user to specify at least one buildpack ID or order ID. This seems like a reasonable configuration to me. Consider that a centralized buildpack registry could be represented as a single CNB package.

Answer 7 · 2019-04-16T03:57:59.000Z

pack create-cnb will package a selection of:

all buildpacks from selected CNB packages,

all <order ID>.toml files from selected CNB packages,

In this proposal, the resulting packaged cnb packages the selected buildpacks from the selected CNB packages?

Answer 8 · 2019-04-16T04:11:10.000Z

Instead of builder.toml, pack create-builder will generate a builder image from a CNB package and stack build image reference.

Would it be possible for the cnbs in the cnb.toml to specify the supported stacks? This would make pack build possible with just a cnb.

Answer 9 · 2019-04-16T04:13:20.000Z

Additionally, it should accept a list of labels via the --label flag. Labels filter such that all specified labels must be present on a group for it to be considered for detection.

👍

Answer 10 · 2019-04-16T04:30:29.000Z

In this proposal, the resulting packaged cnb packages the selected buildpacks from the selected CNB packages?

To rephrase: CNB packages consist of buildpacks and <order ID>.toml files. Because a single buildpack should be distributed as a single-buildpack CNB package, it would be common to source buildpacks and <order ID>.toml files from existing CNB packages when you construct multi-buildpack CNBs.

Would it be possible for the cnbs in the cnb.toml to specify the supported stacks? This would make pack build possible with just a cnb.

Originally, I planned to leave stacks out of the CNB package metadata so that CNB packages are generic (i.e., they can run on any stacks that their buildpacks support). I think you've changed my mind. Not only would including the supported stacks allow CNB packages to be used with pack (and other platforms) in a fully self-contained way, but it would also make it easier to distribute offline (cached) buildpacks.

Answer 11 · 2019-04-16T04:56:07.000Z

I'm not convinced that we want to include the build image, run image, and run image mirrors in each CNB package. What if you combine two CNB packages together and they conflict? Would it be better to leave the image references out, such that CNB + (image references for chosen stack) -> builder?

Answer 12 · 2019-04-17T17:56:24.000Z

Should the value of stacks in the metadata label be an array of stack objects?

Answer 13 · 2019-04-17T18:01:06.000Z

I would also propose that we add the set of packaged buildpacks to the metadata label, as well as the order(s), so that the contents of a cnb on a registry can be inspected without downloading the layer blobs

Answer 14 · 2019-04-18T13:25:20.000Z

How do pack and other platforms know about stacks and where to get them?

Should we have a stack distribution specification?

Answer 15 · 2019-04-24T16:05:53.000Z

Given the discussion at the last WG meeting, I'd like to present two options that integrate this RFC with #3. If we like one of these options, I'll modify this issue accordingly and convert it to a PR.

Option 1

Changes:

"orders" become buildpacks and are defined in a buildpack.toml, just like normal buildpacks
A buildpack is either a single buildpack implementation or an ordering of other buildpack IDs/versions (with no code present).
As orders are now buildpacks, they must have versions.
A single bulidpack.toml can contain any number of bulidpacks.
Apps can use an inline buildpack.toml to define any number of inline buildpacks or additional orderings (of inline or other buildpacks).
Buildpacks can share code/files if they share a buildpack.toml
default version symlinks are only created when the builder is created, and are not included in CNB packages
due to the top-level buildpacks key, buildpack.toml could also serve as an app descriptor file. I'm not sure if there's a strong motivation to do this though, as they seem disjoint to me
labels go away

buildpack.toml example for a single repo that provides a node buildpack, an npm buildpack, and an order definition:

[[buildpacks]]
id = "io.buildpacks.nodejs"
name = "Node.js Buildpack"
version = "0.0.9"

[[buildpacks.order]]
group = [
   { id = "io.buildpacks.node", version = "0.0.5" },
   { id = "io.buildpacks.npm", version = "0.0.7" },
]

[[buildpacks]]
id = "io.buildpacks.npm"
name = "NPM Buildpack"
version = "0.0.7"
path = "./npm-cnb/"
[buildpacks.metadata]
# ...

[[buildpacks]]
id = "io.buildpacks.node"
name = "Node Engine Buildpack"
version = "0.0.5"
path = "./node-cnb/"
[buildpacks.metadata]
# ...

Layer format, where <sha256> = sha256(buildpack.tgz):

/cnb/blobs/<sha256>/
/cnb/by-id/io.buildpacks.nodejs/0.0.9 -> /cnb/blobs/<sha256>/
/cnb/by-id/io.buildpacks.npm/0.0.7 -> /cnb/blobs/<sha256>/
/cnb/by-id/io.buildpacks.node/0.0.5 -> /cnb/blobs/<sha256>/

Option 2

This is the same as option #1, but without allowing multiple buildpacks per buildpack.toml. This is cleaner, but doesn't let you define orders in the same place as buildpacks. This also means inline buildpacks lose the ability to define orders.

Changes:

"orders" become buildpacks and are defined in a buildpack.toml, just like normal buildpacks
A buildpack is either a single buildpack implementation or an ordering of other buildpack IDs/versions (with no code present).
As orders are now buildpacks, they must have versions.
Apps can use an inline buildpack.toml to make themselves buildpacks, but there's no mechanism for specifying orders of other buildpacks.
default version symlinks are only created when the builder is created, and are not included in CNB packages
labels go away

buildpack.toml examples for a node buildpack, an npm buildpack, and an order definition (all separate):

id = "io.buildpacks.nodejs"
name = "Node.js Buildpack"
version = "0.0.9"

[[buildpacks.order]]
group = [
   { id = "io.buildpacks.node", version = "0.0.5" },
   { id = "io.buildpacks.npm", version = "0.0.7" },
]

id = "io.buildpacks.npm"
name = "NPM Buildpack"
version = "0.0.7"
path = "." # default
[metadata]
# ...

id = "io.buildpacks.node"
name = "Node Engine Buildpack"
version = "0.0.5"
path = "." # default
[metadata]
# ...

Layer format (three layers necessary):

/cnb/buildpacks/io.buildpacks.nodejs/0.0.9/

/cnb/buildpacks/io.buildpacks.npm/0.0.7/

/cnb/buildpacks/io.buildpacks.node/0.0.5/

Answer 16 · 2019-05-01T19:29:25.000Z

I'd also like to be able to slice up a buildpack into multiple layers (specifically cached dependencies for offline buildpacks).

Answer 17 · 2019-05-15T07:17:12.000Z

The ordering math looks right to me based on my proposal at summit. Do we lose anything by getting rid of labels?

Answer 18 · 2019-05-16T17:20:15.000Z

From the discussion in the Specification WG, the grouping replaces all the functionality we get from labels, so dropping it for now. If we want to add it back, another RFC can propose it.

Answer 19 · 2019-06-13T03:06:04.000Z

Closing in favor of #11.