Feat: allow creation of layers via stacking (maybe bring back `apply`?)

[smoser]

Is your feature request related to a problem? Please describe.

I would like to be able to publish a set of individual layers and a "combined" layer, where the combined layer is joined layer of the individual layers.

An example, I have a stacker file like this:

base:
  build_only: true
  from:
    type: docker
    url: docker://docker.io/library/ubuntu:jammy
  run:
    apt-get update -q

kernel-build:
  build_only: true
  from:
    type: built
    tag: base
  run: |
      mkdir -p /tmp/stage/kernel /export
      trap "rm -Rf /tmp/stage" EXIT

      apt-get install --no-install-recommends --assume-yes linux-image-virtual
      mv /boot /lib/modules /tmp/stage/kernel
      tar -C /tmp/stage -cf /export/kernel.tar kernel/

# the only top level dir this layer is 'kernel/'
kernel:
  from:
    type: scratch
    url: stacker://kernel-build/export/kernel.tar

ovmf-build:
  build_only: true
  from:
    type: built
    tag: base
  run: |
      mkdir -p /tmp/stage/ovmf /export
      trap "rm -Rf /tmp/stage" EXIT

      apt-get install --no-install-recommends --assume-yes ovmf
      cp /usr/share/OVMF/OVMF_CODE.secboot.fd /tmp/stage/ovmf/ovmf-code.fd
      cp /usr/share/OVMF/OVMF_VARS.fd /tmp/stage/ovmf/ovmf-vars.fd
      tar -C /tmp/stage -cf /export/ovmf.tar ovmf/

# the only top level dir this layer is 'ovmf/'
ovmf:
  from:
    type: scratch
    url: stacker://ovmf-build/export/ovmf.tar

The 'kernel' and 'ovmf' layers are built and have only top level directories 'kernel/' and 'ovmf/' respectively.

I'd like to add another layer build to this file that has:

combined:
  from:
    type: stack
  layers:
    - type: built
      tag: kernel
    - type: built
      tag: ovmf

The goal would be to have 3 layers published:

• ovmf
• kernel
• combined

combined would simply be 'ovmf' on top of 'kernel', such that index.json had:

{
    "schemaVersion": 2,
    "mediaType": "application/vnd.oci.image.index.v1+json",
    "manifests": [
        {
            "mediaType": "application/vnd.oci.image.manifest.v1+json",
            "digest": "sha256:d3743cb7f9799554f6188ce1936259465b06ef181c27ef0ceaf3ce77413ee759",
            "size": 2001,
            "annotations": {
                "org.opencontainers.image.ref.name": "kernel-squashfs"
            }
        },
        {
            "mediaType": "application/vnd.oci.image.manifest.v1+json",
            "digest": "sha256:37f2a392a570e035ef2887918661eca14a984ff2f84d53e5e0d2aaa920900fdf",
            "size": 2000,
            "annotations": {
                "org.opencontainers.image.ref.name": "ovmf-squashfs"
            }
        },
        {
            "mediaType": "application/vnd.oci.image.manifest.v1+json",
            "digest": "sha256:639a0e5badf32904dd4178e6c203fb68884a3b1aa450ce38a44c840321dfbd32",
            "size": 1252,
            "annotations": {
                "org.opencontainers.image.ref.name": "combined-squashfs"
            }
        }
    ]
}

And oci/blobs/sha256/639a0e5badf32904dd4178e6c203fb68884a3b1aa450ce38a44c840321dfbd32 had:

{
    "schemaVersion": 2,
    "mediaType": "application/vnd.oci.image.manifest.v1+json",
    "config": {
        "mediaType": "application/vnd.oci.image.config.v1+json",
        "digest": "sha256:44759f13b3436e793280e2489b53ce8c65c780b81c99763082cdbc3646ab1a3f",
        "size": 376
    },
    "layers": [
      {
          "mediaType": "application/vnd.stacker.image.layer.squashfs+zstd+verity",
          "digest": "sha256:65b132eb797f441c164540aa66e81e7c1d27e45008e82c27487530940335fef4",
          "size": 42680320,
          "annotations": {
              "io.stackeroci.stacker.squashfs_verity_root_hash": "56c9936673423ac92275a9112b6bf0d0f1bbe3fa92767bc519d74484f2ee1936"
          }
      },
      {
          "mediaType": "application/vnd.stacker.image.layer.squashfs+zstd+verity",
          "digest": "sha256:72db3a90a010ee3e954473fabde73f2d05ee25b16615819eef741f7045c3849d",
          "size": 1568768,
          "annotations": {
              "io.stackeroci.stacker.squashfs_verity_root_hash": "3460bb576d5d6568152820b0c0f6f67f2836f46763272717fbc453d6cd245e29"
          }
      }
    ],
    "annotations": {
        "io.stackeroci.stacker.git_version": "v1.0.0-rc4-5-g479fca8",
        "io.stackeroci.stacker.stacker_yaml": "myob",
    }
}

where kernel-squashfs had a single layer in layers with sha of 65b1 and ovmf had a single laye in its layers with sha 72db3

Describe the solution you'd like

No response

Describe alternatives you've considered

No response

Additional context

I'm not set on a particular solution or syntax.

I think this functionality makes sense and can be useful in many contexts. It doesn't work well for traditional package based distributions (with package files spread all over and a single-file package database), but it would work well for non-distro things and potentially even distros like nixOS.

My specific desire is to publish N different layers so that each layer can be pulled individually and a 'combined' layer could be pulled as well without any duplication.

[smoser]

I had originally suggested to @hallyn that maybe this could be implemented as multiple `from' so overloading that (which would require having the yaml parser accept dict or array) we'd have:

combined:
  from:
    - type: built
      tag: kernel
    - type: built
      tag: ovmf

Either solution has to deal with (or ignore) duplicate 'under layers' as discussed in doc/layer-merging.md.

[rchincha]

The following should give us separate layers right?

combined:
  from:
    type: docker
    url: docker://ubuntu:latest
  import:
    - path: docker://...
       dest: /
    - path: stacker://built/...
       dest: /
...

[smoser]

The following should give us separate layers right?

combined:
  from:
    type: docker
    url: docker://ubuntu:latest
  import:
    - path: docker://.../
       dest: /
    - path: stacker://built/.../
       dest: /
...

I edited to add a '/' on the end fof the 'path' elements or clarity based on your comment about trailing / in #453 .

I would expect the above to create a single 'combined' layer that would sit on top of ubuntu:latest. And I think it does, based on existing import behavior, each import does not create its own new layer, does it?