Provide a way to parse out images from rendered manifests and Helm Releases
onedr0p opened this issue · 7 comments
One problem with Helm is that it "hides" images from our GitOps repos, many times I've seen people getting burned by those images not being on the registry yet when they go to merged a PR and Flux tries to reconcile but the image is not available yet.
It would be great if there was a way to parse out the images so we can test if they are available in CI on a PR.
My thinking is that we can leverage the manifest generated by the undocumented feature flux-local get cluster -o yaml which straight up dumps the yaml out.
I think the main design decisions are:
(1) What are the sources of images
(2) What are the changes to the output schema
(3) Any configuration about the output
For (1) I imagine we want everything with a Container ref inside it which I think is mostly the Container and InitContainers field of a PodSpec (orPodTemplateSpec). Not sure if this is possible to get reflectively, or if its just maintaining a list of objects to walk such as these:
- CronJob: spec.jobTemplate.spec.template
- Deployment: spec.template.spec.containers[].image
- ReplicaSet: spec.template.spec.containers[].image
- StatefulSet: spec.template.spec.containers[].image
- DaemonSet: spec.template.spec.containers[].image
- ReplicationController: spec.template.spec.containers[].image
- Pod: spec:containers[].image
Plus initcontainers for all of those.
For (2) i'm thinking either a flat list per kustomization or a list per Kustomization and HelmRelease.
Regarding (3) right now the cluster output is not super customized. For example, we do have HelmChart versions in the schema and we don't output those. It seems like there should be a way to configure how much data is dumped or compacted in the output, but maybe its just an extra binary decision for now.
Input welcome on any of these topics.
The way I am doing this now (number 1) is to have a recursive function that picks out all the image keys in the generated manifests. I am not sure if there is any downsides to doing it this way though.
function extractImageValues(data) {
const imageValues = [];
function extractValues(obj) {
for (const key in obj) {
if (typeof obj[key] === 'object') {
extractValues(obj[key]);
} else if (key === 'image') {
imageValues.push(obj[key]);
}
}
}
extractValues(data);
return imageValues;
}I don't have much input on 2 and 3 so whichever you feel comfortable supporting is fine with me. Also, nice API docs by the way, first time I am seeing that 😄
I've cut 4.1.0 with a first revision of supporting this.
It's a heavyweight command similar to diff in that it will evaluate all helm templates. With the recent performance improvements it takes about 6 seconds on my cluster on my laptop.
Here is an example for how I used it on my cluster:
$ flux-local get cluster --path clusters/prod/ -o yaml --enable-images
---
clusters:
- path: clusters/prod
kustomizations:
- name: certmanager
namespace: flux-system
path: certmanager/prod
helm_repos: []
helm_releases:
- name: cert-manager
namespace: cert-manager
chart:
name: cert-manager
repo_name: jetstack
repo_namespace: flux-system
images:
- quay.io/jetstack/cert-manager-cainjector:v1.13.3
- quay.io/jetstack/cert-manager-controller:v1.13.3
- quay.io/jetstack/cert-manager-ctl:v1.13.3
- quay.io/jetstack/cert-manager-webhook:v1.13.3
cluster_policies: []
- name: crds
namespace: flux-system
path: crds
helm_repos: []
helm_releases: []
cluster_policies: []
- name: flux-system
namespace: flux-system
path: clusters/prod
helm_repos: []
helm_releases: []
cluster_policies: []
images:
- ghcr.io/fluxcd/helm-controller:v0.37.2
- ghcr.io/fluxcd/kustomize-controller:v1.2.1
- ghcr.io/fluxcd/notification-controller:v1.2.3
- ghcr.io/fluxcd/source-controller:v1.2.3
- name: games
namespace: flux-system
path: games/prod
helm_repos: []
helm_releases: []
cluster_policies: []
- name: home
namespace: flux-system
path: home/prod
helm_repos: []
helm_releases:
- name: teslamate
namespace: teslamate
chart:
name: teslamate
repo_name: k8s-at-home
repo_namespace: flux-system
images:
- docker.io/bitnami/postgresql:14.4.0-debian-11-r4
- teslamate/teslamate:1.28.2
- name: mosquitto
namespace: mqtt
chart:
name: mosquitto
repo_name: k8s-at-home
repo_namespace: flux-system
images:
- eclipse-mosquitto:2.0.14
- name: hajimari
namespace: hajimari
chart:
name: hajimari
repo_name: hajimari
repo_namespace: flux-system
images:
- ghcr.io/toboshii/hajimari:v0.3.1
- name: rtsp-to-web
namespace: rtsp-to-web
chart:
name: rtsp-to-web
repo_name: k8s-at-home
repo_namespace: flux-system
images:
- busybox
- ghcr.io/deepch/rtsptoweb:v2.4.3
- name: jellyfin
namespace: jellyfin
chart:
name: app-template
repo_name: bjw-s-helm-charts
repo_namespace: flux-system
images:
- jellyfin/jellyfin:10.8.13
- name: llama-cpp-server
namespace: llama
chart:
name: app-template
repo_name: bjw-s-helm-charts
repo_namespace: flux-system
images:
- busybox:1.36.1
- ghcr.io/allenporter/llama-cpp-server-simple:v1.0.0
- name: llama-cpp-server-cuda
namespace: llama
chart:
name: app-template
repo_name: bjw-s-helm-charts
repo_namespace: flux-system
images:
- busybox:1.36.1
- ghcr.io/allenporter/llama-cpp-server-cuda:v1.0.0
- name: llama-gpt-ui
namespace: llama
chart:
name: app-template
repo_name: bjw-s-helm-charts
repo_namespace: flux-system
images:
- ghcr.io/allenporter/llama-gpt-ui:v0.1.1
cluster_policies: []
- name: infrastructure
namespace: flux-system
path: infrastructure/prod
helm_repos:
- name: podinfo
namespace: flux-system
url: https://stefanprodan.github.io/podinfo
repo_type: default
- name: bitnami
namespace: flux-system
url: oci://registry-1.docker.io/bitnamicharts
repo_type: oci
- name: k8s-at-home
namespace: flux-system
url: https://k8s-at-home.com/charts/
repo_type: default
- name: kubernetes-dashboard
namespace: flux-system
url: https://kubernetes.github.io/dashboard/
repo_type: default
- name: prometheus-community
namespace: flux-system
url: https://prometheus-community.github.io/helm-charts
repo_type: default
- name: kube-state-metrics
namespace: flux-system
url: https://kubernetes.github.io/kube-state-metrics
repo_type: default
- name: rook-release
namespace: flux-system
url: https://charts.rook.io/release
repo_type: default
- name: grafana
namespace: flux-system
url: https://grafana.github.io/helm-charts
repo_type: default
- name: jetstack
namespace: flux-system
url: https://charts.jetstack.io
repo_type: default
- name: haproxytech
namespace: flux-system
url: https://haproxytech.github.io/helm-charts
repo_type: default
- name: itzg
namespace: flux-system
url: https://itzg.github.io/minecraft-server-charts/
repo_type: default
- name: kasten
namespace: flux-system
url: https://charts.kasten.io/
repo_type: default
- name: ingress-nginx
namespace: flux-system
url: https://kubernetes.github.io/ingress-nginx
repo_type: default
- name: benji
namespace: flux-system
url: https://benji-backup.me/helm-charts/
repo_type: default
- name: hajimari
namespace: flux-system
url: https://hajimari.io
repo_type: default
- name: istio
namespace: flux-system
url: https://istio-release.storage.googleapis.com/charts
repo_type: default
- name: nextcloud
namespace: flux-system
url: https://nextcloud.github.io/helm/
repo_type: default
- name: jupyterhub
namespace: flux-system
url: https://jupyterhub.github.io/helm-chart/
repo_type: default
- name: kyverno
namespace: flux-system
url: https://kyverno.github.io/kyverno/
repo_type: default
- name: weave-works
namespace: flux-system
url: https://helm.gitops.weave.works
repo_type: default
- name: bjw-s-helm-charts
namespace: flux-system
url: https://bjw-s.github.io/helm-charts/
repo_type: default
- name: tf-controller
namespace: flux-system
url: https://weaveworks.github.io/tf-controller/
repo_type: default
- name: authelia
namespace: flux-system
url: https://charts.authelia.com
repo_type: default
- name: nvdp
namespace: flux-system
url: https://nvidia.github.io/k8s-device-plugin
repo_type: default
- name: go-skynet
namespace: flux-system
url: https://go-skynet.github.io/helm-charts/
repo_type: default
helm_releases:
- name: redis
namespace: redis
chart:
name: redis
repo_name: bitnami
repo_namespace: flux-system
images:
- docker.io/bitnami/redis:7.0.11-debian-11-r12
- name: rook-ceph
namespace: rook-ceph
chart:
name: rook-ceph
repo_name: rook-release
repo_namespace: flux-system
images:
- rook/ceph:v1.13.0
- name: benji
namespace: benji
chart:
name: benji
repo_name: benji
repo_namespace: flux-system
images:
- docker.io/almalinux:8.7
- docker.io/bitnami/postgresql:14.5.0-debian-11-r35
- ghcr.io/elemental-lf/benji-k8s:0.17.0
- prom/pushgateway:v1.5.1
- name: nvdp
namespace: nvidia-device-plugin
chart:
name: nvidia-device-plugin
repo_name: nvdp
repo_namespace: flux-system
images:
- nvcr.io/nvidia/gpu-feature-discovery:v0.8.2
- nvcr.io/nvidia/k8s-device-plugin:v0.14.3
- registry.k8s.io/nfd/node-feature-discovery:v0.12.1
cluster_policies: []
images:
- docker.io/calico/cni:v3.26.4
- docker.io/calico/kube-controllers:v3.26.4
- docker.io/calico/node:v3.26.4
- postgres:14
- registry.k8s.io/sig-storage/snapshot-controller:v6.3.1
- rook/ceph:master
- name: monitoring
namespace: flux-system
path: monitoring/prod
helm_repos: []
helm_releases:
- name: podinfo-haproxy
namespace: podinfo
chart:
name: podinfo
repo_name: podinfo
repo_namespace: flux-system
images:
- ghcr.io/stefanprodan/podinfo:6.5.4
- name: podinfo-nginx
namespace: podinfo
chart:
name: podinfo
repo_name: podinfo
repo_namespace: flux-system
images:
- ghcr.io/stefanprodan/podinfo:6.5.4
- name: speedtest-prometheus
namespace: monitoring
chart:
name: speedtest-prometheus
repo_name: k8s-at-home
repo_namespace: flux-system
images:
- billimek/prometheus-speedtest-exporter:1.1.0
- name: kubernetes-dashboard
namespace: kubernetes-dashboard
chart:
name: kubernetes-dashboard
repo_name: kubernetes-dashboard
repo_namespace: flux-system
images:
- kubernetesui/dashboard:v2.7.0
- name: kube-prometheus-stack
namespace: monitoring
chart:
name: kube-prometheus-stack
repo_name: prometheus-community
repo_namespace: flux-system
images:
- docker.io/grafana/grafana:10.2.2
- quay.io/kiwigrid/k8s-sidecar:1.25.2
- quay.io/prometheus-operator/prometheus-operator:v0.70.0
- quay.io/prometheus/node-exporter:v1.7.0
- registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20221220-controller-v1.5.1-58-g787ea74b6
- registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.10.1
- name: weave-gitops
namespace: weave-gitops
chart:
name: weave-gitops
repo_name: weave-works
repo_namespace: flux-system
images:
- ghcr.io/weaveworks/wego-app:v0.38.0
cluster_policies: []
- name: network
namespace: flux-system
path: network/prod
helm_repos: []
helm_releases:
- name: unifi
namespace: unifi
chart:
name: unifi
repo_name: k8s-at-home
repo_namespace: flux-system
images:
- jacobalberty/unifi:v7.5.176
- name: adguard-home
namespace: adguard-home
chart:
name: adguard-home
repo_name: k8s-at-home
repo_namespace: flux-system
images:
- adguard/adguardhome:v0.108.0-b.3
- busybox
- name: adguard-home2
namespace: adguard-home
chart:
name: adguard-home
repo_name: k8s-at-home
repo_namespace: flux-system
images:
- adguard/adguardhome:v0.108.0-b.3
- busybox
cluster_policies: []
- name: services
namespace: flux-system
path: services/prod
helm_repos: []
helm_releases:
- name: haproxy
namespace: haproxy
chart:
name: kubernetes-ingress
repo_name: haproxytech
repo_namespace: flux-system
images:
- haproxytech/kubernetes-ingress:1.10.10
- name: haproxy-external
namespace: haproxy
chart:
name: kubernetes-ingress
repo_name: haproxytech
repo_namespace: flux-system
images:
- haproxytech/kubernetes-ingress:1.10.10
- name: metallb
namespace: metallb
chart:
name: metallb
repo_name: bitnami
repo_namespace: flux-system
images:
- docker.io/bitnami/metallb-controller:0.13.10-debian-11-r0
- docker.io/bitnami/metallb-speaker:0.13.9-debian-11-r31
- name: external-dns
namespace: external-dns
chart:
name: external-dns
repo_name: bitnami
repo_namespace: flux-system
images:
- docker.io/bitnami/external-dns:0.13.4-debian-11-r19
cluster_policies: []
- name: settings
namespace: flux-system
path: settings/prod
helm_repos: []
helm_releases: []
cluster_policies: []
Looks good @allenporter, looking at this quick it seems like we can create a new GH workflow and pass the arg --enable-images arg and then do processing on the output?
Yes, that is how i was expecting it would be used.
Are you suggesting it would be useful to have a shared github action similar to diff and test that dumps the manifest output somewhere?
That would be neat but I'm sure I could glue it together in a workflow too.
OK great. The other thing I do today is run it as a pre-commit but this won't really help for updating cluster images updated as a pre-commit: https://github.com/allenporter/k8s-gitops/blob/main/scripts/update-manifest.sh
I don't currently do anything with that output, but the idea was that it could be used for asserting things about the cluster (e.g. certain releases are present, etc). Maybe it could be updated in a workflow out of band.
Anyway, play with it and we can see if there are future improvements or ways to make this useful.