grafana/loki

CI: Helm chart publishing still broken

Opened this issue · 5 comments

trevorwhitney commented

#14799 fixed the helm-ci action, allowing helm PRs to be merged. However, publishing releases is still broken:

Run helm push "/home/runner/work/loki/loki/.cr-release-packages/loki-6.19.0.tgz" "oci://ghcr.io/${GITHUB_REPOSITORY_OWNER}/helm-charts"
  helm push "/home/runner/work/loki/loki/.cr-release-packages/loki-6.19.0.tgz" "oci://ghcr.io/${GITHUB_REPOSITORY_OWNER}/helm-charts"
  shell: /usr/bin/bash -e {0}
  env:
    CR_CONFIGFILE: /home/runner/work/loki/loki/source/production/helm/cr.yaml
    CT_CONFIGFILE: /home/runner/work/loki/loki/source/production/helm/ct.yaml
    CR_INDEX_PATH: /home/runner/work/loki/loki/.cr-index
    CR_PACKAGE_PATH: /home/runner/work/loki/loki/.cr-release-packages
    CR_TOOL_PATH: /home/runner/work/loki/loki/.cr-tool
    HELM_TAG_PREFIX: helm
    github_app_id: ***
    AUTHTOKEN: ***
Error: unexpected status from POST request to https://ghcr.io/v[2](https://github.com/grafana/loki/actions/runs/11727098884/job/32667387047#step:18:2)/grafana/helm-charts/loki/blobs/uploads/: 403 Forbidden
Error: Process completed with exit code 1.

Last successful run was 3 weeks ago: https://github.com/grafana/loki/actions/runs/11374046600

trevorwhitney commented

@morganrowse if you're still interested, looks like the saga isn't quite over

morganrowse commented

Hi @trevorwhitney just got back from leave and now ready to look into this.

I have a theory that this CI issue was introduced by some upstream changes to grafana/helm-charts shared github actions here, The grafana/loki helm-release action makes use of the main branch here, something akin to using latest on a docker container.

It looks like the core issue here follows this PR grafana/helm-charts#3418

@narqo just raising for visibility if you have not seen this for the Loki chart. Im not seeing any fundemental code differences between mimr and loki CI. Are there GitHub app or repo permission settings required to get this working as expected?

grafana/helm-charts#3068 (comment)

narqo commented

Im not seeing any fundemental code differences between mimr and loki CI. Are there GitHub app or repo permission settings required to get this working as expected?

It should just work since grafana/helm-charts#3424 — no special changes on your side needed (it works for k6-operator also).

trevorwhitney commented

@narqo we're referencing thew workflow @ main but still seeing those 403 errors Error: unexpected status from POST request to https://ghcr.io/v2/grafana/helm-charts/loki/blobs/uploads/: 403 Forbidden

trevorwhitney commented

Awesome, thanks @morganrowse, I reverted the release pipeline to 70dbbb722dee3f2ee126e12684cc0e92a20972ed before the ghcr changes, and our release went out successfully this week. Going to keep this issue open until we fix it for real though, as we should bring back the ghcr pushing.