goharbor/harbor-helm

Cannot use existingSecret for external redis

CrimsonFez opened this issue · 9 comments

CrimsonFez commented

When I configure an externalSecret from external redis I have the following errors:

upgrade.go:144: [debug] preparing upgrade for harbor
Error: UPGRADE FAILED: YAML parse error on harbor/templates/jobservice/jobservice-cm.yaml: error converting YAML to JSON: yaml: line 20: found character that cannot start any token
helm.go:84: [debug] error converting YAML to JSON: yaml: line 20: found character that cannot start any token
YAML parse error on harbor/templates/jobservice/jobservice-cm.yaml
helm.sh/helm/v3/pkg/releaseutil.(*manifestFile).sort
	/builddir/build/BUILD/helm-3.11.1/_build/src/helm.sh/helm/v3/pkg/releaseutil/manifest_sorter.go:146
helm.sh/helm/v3/pkg/releaseutil.SortManifests
	/builddir/build/BUILD/helm-3.11.1/_build/src/helm.sh/helm/v3/pkg/releaseutil/manifest_sorter.go:106
helm.sh/helm/v3/pkg/action.(*Configuration).renderResources
	/builddir/build/BUILD/helm-3.11.1/_build/src/helm.sh/helm/v3/pkg/action/action.go:170
helm.sh/helm/v3/pkg/action.(*Upgrade).prepareUpgrade
	/builddir/build/BUILD/helm-3.11.1/_build/src/helm.sh/helm/v3/pkg/action/upgrade.go:236
helm.sh/helm/v3/pkg/action.(*Upgrade).RunWithContext
	/builddir/build/BUILD/helm-3.11.1/_build/src/helm.sh/helm/v3/pkg/action/upgrade.go:145
main.newUpgradeCmd.func2
	/builddir/build/BUILD/helm-3.11.1/_build/src/helm.sh/helm/v3/cmd/helm/upgrade.go:201
github.com/spf13/cobra.(*Command).execute
	/usr/share/gocode/src/github.com/spf13/cobra/command.go:940
github.com/spf13/cobra.(*Command).ExecuteC
	/usr/share/gocode/src/github.com/spf13/cobra/command.go:1068
github.com/spf13/cobra.(*Command).Execute
	/usr/share/gocode/src/github.com/spf13/cobra/command.go:992
main.main
	/builddir/build/BUILD/helm-3.11.1/_build/src/helm.sh/helm/v3/cmd/helm/helm.go:83
runtime.main
	/usr/lib/golang/src/runtime/proc.go:267
runtime.goexit
	/usr/lib/golang/src/runtime/asm_amd64.s:1650
UPGRADE FAILED
main.newUpgradeCmd.func2
	/builddir/build/BUILD/helm-3.11.1/_build/src/helm.sh/helm/v3/cmd/helm/upgrade.go:203
github.com/spf13/cobra.(*Command).execute
	/usr/share/gocode/src/github.com/spf13/cobra/command.go:940
github.com/spf13/cobra.(*Command).ExecuteC
	/usr/share/gocode/src/github.com/spf13/cobra/command.go:1068
github.com/spf13/cobra.(*Command).Execute
	/usr/share/gocode/src/github.com/spf13/cobra/command.go:992
main.main
	/builddir/build/BUILD/helm-3.11.1/_build/src/helm.sh/helm/v3/cmd/helm/helm.go:83
runtime.main
	/usr/lib/golang/src/runtime/proc.go:267
runtime.goexit
	/usr/lib/golang/src/runtime/asm_amd64.s:1650

From what I can tell this is actually an issue with b64dec from helm.

In _heplers.tpl on line 161 it grabs the existing secret data and decodes it to use when making the config map.

{{- define "harbor.redis.pwdfromsecret" -}}
  {{- (lookup "v1" "Secret"  .Release.Namespace (.Values.redis.external.existingSecret)).data.REDIS_PASSWORD | b64dec -}}
{{- end -}}

At one point in my testing I was also able to generate the manifest and everything looked fine, the password was in the url, but it still threw the error.
After I replace the lookup with print "base64string" | b64dec it still caused the error. I also dont believe that it was an issue with my password since it works just fine if I do print "password".

Kajot-dev commented

Duplicate of #1641
See my comment there for exact explanation

CrimsonFez commented

This is not a duplicate because I'm not using helm template, I'm using helm upgrade/install.

Kajot-dev commented

Sorry, you're right. Can you render the invalid YAML and paste it here? (you should be able with --debug option)

CrimsonFez commented

It doesn't output any yaml. Just the error

❯ helm -n harbor upgrade --install harbor harbor/harbor --version 1.14.1 --values values.yaml --debug
history.go:56: [debug] getting history for release harbor
upgrade.go:144: [debug] preparing upgrade for harbor
Error: UPGRADE FAILED: YAML parse error on harbor/templates/jobservice/jobservice-cm.yaml: error converting YAML to JSON: yaml: line 20: found character that cannot start any token
helm.go:84: [debug] error converting YAML to JSON: yaml: line 20: found character that cannot start any token
YAML parse error on harbor/templates/jobservice/jobservice-cm.yaml
helm.sh/helm/v3/pkg/releaseutil.(*manifestFile).sort
	/builddir/build/BUILD/helm-3.11.1/_build/src/helm.sh/helm/v3/pkg/releaseutil/manifest_sorter.go:146
helm.sh/helm/v3/pkg/releaseutil.SortManifests
	/builddir/build/BUILD/helm-3.11.1/_build/src/helm.sh/helm/v3/pkg/releaseutil/manifest_sorter.go:106
helm.sh/helm/v3/pkg/action.(*Configuration).renderResources
	/builddir/build/BUILD/helm-3.11.1/_build/src/helm.sh/helm/v3/pkg/action/action.go:170
helm.sh/helm/v3/pkg/action.(*Upgrade).prepareUpgrade
	/builddir/build/BUILD/helm-3.11.1/_build/src/helm.sh/helm/v3/pkg/action/upgrade.go:236
helm.sh/helm/v3/pkg/action.(*Upgrade).RunWithContext
	/builddir/build/BUILD/helm-3.11.1/_build/src/helm.sh/helm/v3/pkg/action/upgrade.go:145
main.newUpgradeCmd.func2
	/builddir/build/BUILD/helm-3.11.1/_build/src/helm.sh/helm/v3/cmd/helm/upgrade.go:201
github.com/spf13/cobra.(*Command).execute
	/usr/share/gocode/src/github.com/spf13/cobra/command.go:940
github.com/spf13/cobra.(*Command).ExecuteC
	/usr/share/gocode/src/github.com/spf13/cobra/command.go:1068
github.com/spf13/cobra.(*Command).Execute
	/usr/share/gocode/src/github.com/spf13/cobra/command.go:992
main.main
	/builddir/build/BUILD/helm-3.11.1/_build/src/helm.sh/helm/v3/cmd/helm/helm.go:83
runtime.main
	/usr/lib/golang/src/runtime/proc.go:267
runtime.goexit
	/usr/lib/golang/src/runtime/asm_amd64.s:1650
UPGRADE FAILED
main.newUpgradeCmd.func2
	/builddir/build/BUILD/helm-3.11.1/_build/src/helm.sh/helm/v3/cmd/helm/upgrade.go:203
github.com/spf13/cobra.(*Command).execute
	/usr/share/gocode/src/github.com/spf13/cobra/command.go:940
github.com/spf13/cobra.(*Command).ExecuteC
	/usr/share/gocode/src/github.com/spf13/cobra/command.go:1068
github.com/spf13/cobra.(*Command).Execute
	/usr/share/gocode/src/github.com/spf13/cobra/command.go:992
main.main
	/builddir/build/BUILD/helm-3.11.1/_build/src/helm.sh/helm/v3/cmd/helm/helm.go:83
runtime.main
	/usr/lib/golang/src/runtime/proc.go:267
runtime.goexit
	/usr/lib/golang/src/runtime/asm_amd64.s:1650
zyyw commented

@CrimsonFez could you please share with us which version of harbor-helm you are using when facing this issue? Thanks

CrimsonFez commented

1.14.1

MinerYang commented

Hi @CrimsonFez ,

  • Could you try to upgrade/helm using redis password instead of existingSecret to narrowdown the error scope
  • Please share your values.yaml and check if you quote the fields.
CrimsonFez commented

I currently deploy with the password in my values, so that works.
This issue is present on 1.14.0, 1.14.1, and 1.14.2.

Here are my values:

expose:
  tls:
    certSource: secret
    secret:
      secretName: harbor-ingress
  ingress:
    hosts:
      core: harbor.example.com
    harbor:
      annotations:
        cert-manager.io/cluster-issuer: letsencrypt-prod-http

externalURL: https://harbor.example.com

core:
  replicas: 2

registry:
  replicas: 2

portal:
  replicas: 2

persistence:
  resourcePolicy: "keep"
  persistentVolumeClaim:
    trivy:
      storageClass: "rbd-ssd-r3"
    registry:
      storageClass: "cephfs-fast"
      accessMode: "ReadWriteMany"
      size: "50Gi"
  imageChartStorage:
    type: filesystem
    disableredirect: true
    filesystem:
      rootdirectory: /storage
      maxthreads: 100

database:
  type: external
  external:
    host: harbor-pg-primary
    username: harbor
    coreDatabase: harbor
    existingSecret: harbor-pg-pguser-harbor
    sslmode: "require"

redis:
  type: external
  external:
    addr: keydb:6379
    existingSecret: redis-password

jobservice:
  jobLoggers:
    - database

logLevel: error
Vertiwell commented

I have the same issue, install redis with the bitnami chart, install with the custom values:

auth:
  enabled: true
  sentinel: true
  existingSecret: "redis-secret"
  existingSecretPasswordKey: "REDIS_PASSWORD"

copy secret to harbor namespace, install harbor chart with:

redis:
  type: external
  external:
    addr: "redis-node-0.redis-headless.redis.svc.cluster.local:26379......"
    sentinelMasterSet: "mymaster"
    existingSecret: "redis-secret"

Errors:

**sentinel.go:514: sentinel: GetMasterAddrByName master="mymaster" failed: NOAUTH Authentication required.
failed to ping redis+sentinel://:xxxxx@redis-node-0.redis-headless.redis.svc.cluster.local:26379**

Change the redis chart to:

auth:
  enabled: true
  sentinel: false

make no other changes, and Harbor starts working straight away:
sentinel.go:661: sentinel: new master="mymaster" addr="redis-node-0.redis-headless.redis.svc.cluster.local:6379"

I would have expected it to fail since I'm still telling Harbor to provide a password in the Helm chart with existingSecret.
If I run:
kubectl exec -it redis-cli-pod -- redis-cli -h redis-node-0.redis-headless.redis.svc.cluster.local -p 26379 -a $REDIS_PASSWORD SENTINEL get-master-addr-by-name mymaster

Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
AUTH failed: ERR AUTH <password> called without any password configured for the default user. Are you sure your configuration is correct?
1) "redis-node-3.redis-headless.redis.svc.cluster.local"
2) "6379"