Klipper is unable to upgrade chart with PDB

Question

Klipper is unable to upgrade chart with PDB

sergeyshaykhullin opened this issue a year ago · 12 comments

cockroachdb/helm-charts#270 (comment)
https://github.com/k3s-io/klipper-helm/blob/5e763524fd8cab4790289e7220e04a6d2a016adc/entry#L63

Install logs on k3s 1.26.3

if [[ ${KUBERNETES_SERVICE_HOST} =~ .*:.* ]]; then
	echo "KUBERNETES_SERVICE_HOST is using IPv6"
	CHART="${CHART//%\{KUBERNETES_API\}%/[${KUBERNETES_SERVICE_HOST}]:${KUBERNETES_SERVICE_PORT}}"
else
	CHART="${CHART//%\{KUBERNETES_API\}%/${KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT}}"
fi

set +v -x
+ [[ '' != \t\r\u\e ]]
+ export HELM_HOST=127.0.0.1:44134
+ HELM_HOST=127.0.0.1:44134
+ helm_v2 init --skip-refresh --client-only --stable-repo-url https://charts.helm.sh/stable/
+ tiller --listen=127.0.0.1:44134 --storage=secret
Creating /home/klipper-helm/.helm 
Creating /home/klipper-helm/.helm/repository 
Creating /home/klipper-helm/.helm/repository/cache 
Creating /home/klipper-helm/.helm/repository/local 
Creating /home/klipper-helm/.helm/plugins 
Creating /home/klipper-helm/.helm/starters 
Creating /home/klipper-helm/.helm/cache/archive 
Creating /home/klipper-helm/.helm/repository/repositories.yaml 
Adding stable repo with URL: https://charts.helm.sh/stable/ 
Adding local repo with URL: http://127.0.0.1:8879/charts 
$HELM_HOME has been configured at /home/klipper-helm/.helm.
Not installing Tiller due to 'client-only' flag having been set
++ timeout -s KILL 30 helm_v2 ls --all '^crdb$' --output json++ 
jq -r '.Releases | length'
[main] 2023/03/30 19:41:24 Starting Tiller v2.17.0 (tls=false)
[main] 2023/03/30 19:41:24 GRPC listening on 127.0.0.1:44134
[main] 2023/03/30 19:41:24 Probes listening on :44135
[main] 2023/03/30 19:41:24 Storage driver is Secret
[main] 2023/03/30 19:41:24 Max history per release is 0
[storage] 2023/03/30 19:41:24 listing all releases with filter
+ V2_CHART_EXISTS=
+ [[ '' == \1 ]]
+ [[ v3 == \v\2 ]]
+ [[ -f /config/ca-file.pem ]]
+ [[ -n '' ]]
+ shopt -s nullglob
+ helm_content_decode
+ set -e
+ ENC_CHART_PATH=/chart/crdb.tgz.base64
+ CHART_PATH=/tmp/crdb.tgz
+ [[ ! -f /chart/crdb.tgz.base64 ]]
+ return
+ [[ template != \d\e\l\e\t\e ]]
+ helm_repo_init
+ grep -q -e 'https\?://'
+ [[ helm_v3 == \h\e\l\m\_\v\3 ]]
+ [[ cockroachdb == stable/* ]]
+ [[ -n https://charts.cockroachdb.com ]]
+ helm_v3 repo add crdb https://charts.cockroachdb.com
"crdb" has been added to your repositories
+ helm_v3 repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "crdb" chart repository
Update Complete. ⎈Happy Helming!⎈
+ helm_update template --namespace crdb --repo https://charts.cockroachdb.com --version 10.0.7 --debug
+ [[ helm_v3 == \h\e\l\m\_\v\3 ]]
++ helm_v3 ls --all -f '^crdb$' --namespace crdb --output json
++ jq -r '"\(.[0].app_version),\(.[0].status)"'
++ tr '[:upper:]' '[:lower:]'
+ LINE=22.1.6,deployed
+ IFS=,
+ read -r INSTALLED_VERSION STATUS _
+ VALUES=
+ for VALUES_FILE in /config/*.yaml
+ VALUES=' --values /config/values-01_HelmChart.yaml'
+ [[ template = \d\e\l\e\t\e ]]
+ [[ 22.1.6 =~ ^(|null)$ ]]
+ [[ deployed =~ ^(pending-install|pending-upgrade|pending-rollback)$ ]]
+ [[ deployed == \d\e\p\l\o\y\e\d ]]
+ echo 'Already installed crdb'
Already installed crdb
+ [[ helm_v3 == \h\e\l\m\_\v\3 ]]
+ helm_v3 mapkubeapis crdb --namespace crdb
2023/03/30 19:41:25 Release 'crdb' will be checked for deprecated or removed Kubernetes APIs and will be updated if necessary to supported API versions.
2023/03/30 19:41:25 Get release 'crdb' latest version.
2023/03/30 19:41:25 Check release 'crdb' for deprecated or removed APIs...
2023/03/30 19:41:25 Finished checking release 'crdb' for deprecated or removed APIs.
2023/03/30 19:41:25 Release 'crdb' has no deprecated or removed APIs.
2023/03/30 19:41:25 Map of release 'crdb' deprecated or removed APIs to supported versions, completed successfully.
+ echo 'Upgrading helm_v3 chart'
+ echo 'Upgrading crdb'
Upgrading crdb
+ shift 1
+ helm_v3 upgrade --namespace crdb --repo https://charts.cockroachdb.com --version 10.0.7 --debug crdb cockroachdb --values /config/values-01_HelmChart.yaml
upgrade.go:142: [debug] preparing upgrade for crdb
upgrade.go:150: [debug] performing update for crdb
Error: UPGRADE FAILED: unable to build kubernetes objects from current release manifest: resource mapping not found for name: "crdb-budget" namespace: "crdb" from "": no matches for kind "PodDisruptionBudget" in version "policy/v1beta1"
ensure CRDs are installed first
helm.go:84: [debug] resource mapping not found for name: "crdb-budget" namespace: "crdb" from "": no matches for kind "PodDisruptionBudget" in version "policy/v1beta1"
ensure CRDs are installed first
unable to build kubernetes objects from current release manifest
helm.sh/helm/v3/pkg/action.(*Upgrade).performUpgrade
	helm.sh/helm/v3/pkg/action/upgrade.go:273
helm.sh/helm/v3/pkg/action.(*Upgrade).RunWithContext
	helm.sh/helm/v3/pkg/action/upgrade.go:151
main.newUpgradeCmd.func2
	helm.sh/helm/v3/cmd/helm/upgrade.go:199
github.com/spf13/cobra.(*Command).execute
	github.com/spf13/cobra@v1.5.0/command.go:872
github.com/spf13/cobra.(*Command).ExecuteC
	github.com/spf13/cobra@v1.5.0/command.go:990
github.com/spf13/cobra.(*Command).Execute
	github.com/spf13/cobra@v1.5.0/command.go:918
main.main
	helm.sh/helm/v3/cmd/helm/helm.go:83
runtime.main
	runtime/proc.go:250
runtime.goexit
	runtime/asm_amd64.s:1571
UPGRADE FAILED
main.newUpgradeCmd.func2
	helm.sh/helm/v3/cmd/helm/upgrade.go:201
github.com/spf13/cobra.(*Command).execute
	github.com/spf13/cobra@v1.5.0/command.go:872
github.com/spf13/cobra.(*Command).ExecuteC
	github.com/spf13/cobra@v1.5.0/command.go:990
github.com/spf13/cobra.(*Command).Execute
	github.com/spf13/cobra@v1.5.0/command.go:918
main.main
	helm.sh/helm/v3/cmd/helm/helm.go:83
runtime.main
	runtime/proc.go:250
runtime.goexit
	runtime/asm_amd64.s:1571

Answer 1 · 2023-03-30T19:52:11.000Z

Error: UPGRADE FAILED: unable to build kubernetes objects from current release manifest: resource mapping not found for name: "crdb-budget" namespace: "crdb" from "": no matches for kind "PodDisruptionBudget" in version "policy/v1beta1"
ensure CRDs are installed first
helm.go:84: [debug] resource mapping not found for name: "crdb-budget" namespace: "crdb" from "": no matches for kind "PodDisruptionBudget" in version "policy/v1beta1"

https://kubernetes.io/docs/reference/using-api/deprecation-guide/#poddisruptionbudget-v125

It looks like this was just fixed in the helm-mapkubeapis plugin 3 weeks ago: https://github.com/helm/helm-mapkubeapis/pull/102/files

If you want to upgrade this chart prior to our pulling in that update to the API mapping plugin, you'll need to install helm and the plugin on an administrative host and perform the migration manually.

Answer 2 · 2023-05-30T20:33:51.000Z

@brandond Does this patch included in 1.27.2+k3s1?

Already installed crdb
2023/05/30 20:31:50 Release 'crdb' will be checked for deprecated or removed Kubernetes APIs and will be updated if necessary to supported API versions.
2023/05/30 20:31:50 Get release 'crdb' latest version.
2023/05/30 20:31:50 Check release 'crdb' for deprecated or removed APIs...
2023/05/30 20:31:50 Finished checking release 'crdb' for deprecated or removed APIs.
2023/05/30 20:31:50 Release 'crdb' has no deprecated or removed APIs.
2023/05/30 20:31:50 Map of release 'crdb' deprecated or removed APIs to supported versions, completed successfully.
+ echo 'Upgrading helm_v3 chart'
+ echo 'Upgrading crdb'
Upgrading crdb
+ shift 1
+ helm_v3 upgrade --namespace crdb --version 11.0.1 crdb crdb/cockroachdb --values /config/values-01_HelmChart.yaml
Error: UPGRADE FAILED: unable to build kubernetes objects from current release manifest: resource mapping not found for name: "crdb-budget" namespace: "crdb" from "": no matches for kind "PodDisruptionBudget" in version "policy/v1beta1"
ensure CRDs are installed first

Answer 3 · 2023-05-30T21:48:32.000Z

Yes it is, but note that it only handles migrating existing resources. If the chart is trying to add new resources using the wrong API version, the mapkubeapis plugin won't handle that - you need to update the chart itself. Can you confirm that the chart is using the new version?

Answer 4 · 2023-06-01T09:22:58.000Z

@brandond Yes, crdb chart should work with 1.25+

cockroachdb/helm-charts#270 (comment)

Answer 5 · 2023-06-01T09:37:04.000Z

What version of the chart do you currently have installed, and what version are you upgrading to? Can you share the HelmChart yaml?

Answer 6 · 2023-06-01T09:46:47.000Z

@brandond helm_v3 upgrade --namespace crdb --version 11.0.1 crdb crdb/cockroachdb --values /config/values-01_HelmChart.yaml

k3s 1.24 -> 1.27
crdb 8.1.5 -> 11.0.1

chart: cockroachdb
helmVersion: v3
repo: https://charts.cockroachdb.com
targetNamespace: crdb
valuesContent: |
  nameOverride: crdb
  fullnameOverride: crdb

  conf:
    cache: 10%
    max-sql-memory: 30%
    log:
      enabled: true
      config:
        file-defaults:
          format: json
    store:
      enabled: true

  init:
    provisioning:
      enabled: true
      users: []
      databases: []

  statefulset:
    replicas: REDACTED
    podManagementPolicy: OrderedReady

    annotations:
      prometheus.io/scrape: "true"
      prometheus.io/port: "8080"
      prometheus.io/path: _status/vars

    resources:
      limits:
        cpu: REDACTED
        memory: REDACTED
      requests:
        cpu: REDACTED
        memory: REDACTED

    podAntiAffinity:
      topologyKey: kubernetes.io/hostname
      type: hard

    customLivenessProbe:
      httpGet:
        path: /health
        port: http
      initialDelaySeconds: 30
      periodSeconds: 30
      timeoutSeconds: 20
    customReadinessProbe:
      httpGet:
        path: /health
        port: http
      initialDelaySeconds: 30
      periodSeconds: 30
      timeoutSeconds: 20

    env:
      - name: GOMEMLIMIT
        value: REDACTED

  ingress:
    enabled: true
    annotations:
      kubernetes.io/tls-acme: "true"
      nginx.ingress.kubernetes.io/auth-url: https://$host/oauth2/auth
      nginx.ingress.kubernetes.io/auth-signin: https://$host/oauth2/start?rd=$escaped_request_uri
      nginx.ingress.kubernetes.io/service-upstream: "true"
    hosts:
      - REDACTED
    tls:
      - hosts:
          - REDACTED
        secretName: crdb-tls

  storage:
    persistentVolume:
      enabled: true
      size: REDACTED
      storageClass: REDACTED

  tls:
    enabled: false
    certs:
      selfSigner:
        enabled: false
version: 11.0.1

Answer 7 · 2023-06-01T09:52:31.000Z

Helm template version is ok

# Source: cockroachdb/templates/poddisruptionbudget.yaml
kind: PodDisruptionBudget
apiVersion: policy/v1
metadata:
  name: crdb-budget
  namespace: "crdb"
  labels:
    helm.sh/chart: cockroachdb-11.0.1
    app.kubernetes.io/name: crdb
    app.kubernetes.io/instance: "crdb"
    app.kubernetes.io/managed-by: "Helm"
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: crdb
      app.kubernetes.io/instance: "crdb"
      app.kubernetes.io/component: cockroachdb
  maxUnavailable: 1

Answer 8 · 2023-06-01T19:15:38.000Z

Did you go directly from K3s 1.24 to 1.27 without stepping through 1.25 and 1.26?

Also, why are you redacting your resource requests/limits, replicas, and so on? I'd like to test this HelmChart configuration locally but the redaction of inconsequential values makes that difficult.

Answer 9 · 2023-06-01T19:26:44.000Z

We had k3s 1.24 + crdb 8.1.5

Then bump k3s to 1.25, 1.26, 1.27, but because of broken pdb crdb chart was still 8.1.5 and we redact crdb sts image version manually for a long time

Now we tried to upgrade crdb chart version with fixed pdb, but it is still failing

It is not so important, you can remove requests/limits and set replicas to 3

Answer 10 · 2023-06-01T19:53:56.000Z

helm-mapkubeapis is unable to update the resources because the resources are templated in an unexpected way. The standard is to put the apiVersion before the kind:

apiVersion: policy/v1beta1
kind: PodDisruptionBudget

but the crdb chart has:

kind: PodDisruptionBudget
apiVersion: policy/v1beta1

Ref: https://github.com/cockroachdb/helm-charts/blob/aa91c3fc0c453c8ec9fac411c8ddd886f6ffc720/cockroachdb/templates/poddisruptionbudget.yaml

This is documented in the plugin: https://github.com/helm/helm-mapkubeapis#api-mapping

The search and replace strings are in order with apiVersion first and then kind. This should be changed if the Helm release metadata is rendered with different search/replace string.

Answer 11 · 2023-06-01T19:55:16.000Z

You'll need to install the helm-mapkubeapis plugin yourself, modify the mapping file at https://github.com/helm/helm-mapkubeapis/blob/main/config/Map.yaml#L114-L115 to handle the reversed attribute order, and then execute it to migrate the versions in your existing chart deployment.

Answer 12 · 2023-06-01T20:35:13.000Z

@brandond Thank you! This fixed upgrade problem