Kernel publish job is having an issue

Question

Kernel publish job is having an issue

Closed this issue 2 years ago · 9 comments

Expected Behaviour

Kernel publish job is showing success, but is failing to upload the multi-arch image to quay. Need to troubleshoot further: https://github.com/tinkerbell/hook/actions/runs/558935852

Answer 1 · 2021-02-17T14:41:43.000Z

I believe the issue is related to this buildx issue: docker/buildx#177

Possible workaround may be to use a local registry for the build portion like we are doing for the kernel CI job, and then add an additional step to move the image to quay.io. This would also give us the opportunity to eventually do additional validation of the image prior to publishing in the future if we wish.

Answer 2 · 2021-02-17T15:19:04.000Z

After reviewing the kernel CI job, it looks like my previous suggestion may not work...

50 merging manifest list 147.75.101.109:5000/hook-kernel:5.10.11-78c072732359e42a1083c5801516fb2a2ca4c30c
#50 sha256:da8d1bc099b9e76087b416fcaccb5599e3b62e4e0a762b236aa9e14feb9cdc8f
#50 ERROR: httpReadSeeker: failed open: failed to do request: Get https://147.75.101.109:5000/v2/hook-kernel/manifests/sha256:ec951534d60a024cd7d3130a13a532d6dad0186c8a0317789871ce516dc95099: http: server gave HTTP response to HTTPS client

Might need to choose one of the following approaches, instead:

Build/Tag the arch-specific images separately and combine them using docker manifest or using docker buildx imagetools
- If we go this route we might want to break up the kernel ci/push jobs into multiple jobs so that the amd64 and arm64 builds are done in parallel and the step to combine them in a separate job that is configured with the build jobs as dependencies: https://docs.github.com/en/actions/learn-github-actions/managing-complex-workflows#creating-dependent-jobs
Export the manifest list image to a different format rather than pushing directly to a registry
- The exported manifest list image would need to be imported into the registry in some form, it might be possible to do this in a single step using docker buildx imagetools, otherwise it will be necessary to push the individual manifests for each platform and then create the manifest list.

Answer 3 · 2021-02-17T16:16:00.000Z

Digging a bit further, buildx output type of oci and tar do not work with multi-node builds. We might be able to work with an output type of local, I'm still testing with that approach.

Answer 4 · 2021-02-17T16:50:21.000Z

Client-side manifest merging issue for the local registry is here: docker/buildx#354

Answer 5 · 2021-03-18T14:51:06.000Z

Once we are managing CI runners with https://github.com/tinkerbell/infrastructure, we can update the kernel jobs to build arch-specific builds on the appropriate runners and then build the manifest image after the arch-specific images are complete.

Answer 6 · 2021-08-27T03:50:30.000Z

@detiber - was this ever solved?

Answer 7 · 2022-02-08T16:11:32.000Z

@detiber Just pinging you again about whether it got solved.

Answer 8 · 2022-02-08T16:45:55.000Z

https://github.com/tinkerbell/hook/actions/runs/1813512375 will build and push a new kernel, if its successful we should close this.

Answer 9 · 2022-02-15T16:11:07.000Z

Looking at quay it looks like the builds are being pushed just fine so I'm marking this as fixed/closing.