Pass helm credentials

We have added a private helm repository to our flux deployment.
Flux gets its credentials from a secret in the kubernetes cluster (which is therefore of course not available to flux-local in the repo).
Example helm repository:

---
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: HelmRepository
[...]
  url: https://gitlab.example.com/api/v4/projects/1234/packages/helm/stable
  secretRef:
    name: helm-pull-token

Accordingly, a flux-local run in the CI pipeline terminates with the following error:

Command '(None) helm template clusterconfig flux-system-clusterconfig/clusterconfig --namespace clusterconfig --skip-crds --skip-tests --version 0.1.0 --registry-config /dev/null --repository-cache /tmp/tmp0nzo4u72 --repository-config /tmp/tmpiynunb2i/repository-config.yaml' failed with return code 1
Error: no cached repo found. (try 'helm repo update'): open /tmp/tmp0nzo4u72/flux-system-clusterconfig-index.yaml: no such file or directory
flux-local error:  Command '(None) helm template clusterconfig flux-system-clusterconfig/clusterconfig --namespace clusterconfig --skip-crds --skip-tests --version 0.1.0 --registry-config /dev/null --repository-cache /tmp/tmp0nzo4u72 --repository-config /tmp/tmpiynunb2i/repository-config.yaml' failed with return code 1
Error: no cached repo found. (try 'helm repo update'): open /tmp/tmp0nzo4u72/flux-system-clusterconfig-index.yaml: no such file or directory

We would therefore need a way to also provide a secret via flux-local to helm, so that we could pull and diff the private helmrelease from within a ci pipeline.

Hi, I'm not sure that error is related to the private secret. It seems more like it's just having a problem managing helm local cache directory? Perhaps you could run with --log-level=DEBUG (before other params) and pull out any relevant helm lines? It's supposed to make all the needed directories for helm.

Oh I see, it can't pull the helm chart from the repo, you are right. I misunderstood thinking we were talking about the release, not repo.

Do you have an example of how to use helm cli with secrets like this? Are there fields in the repository config or something else?

Hi @allenporter, sorry for the late response.
I build a minimal setup with just one Kustomization with a HelmRepo and HelmRelease from the private HelmChart.
These are the complete logs:

$ flux-local  --log-level=DEBUG   build  --enable-helm --skip-crds .

DEBUG:asyncio:Using selector: KqueueSelector
DEBUG:flux_local.git_repo:Processing cluster with selector ResourceSelector(path=PathSelector(path=PosixPath('.'), sources=None), cluster=MetadataSelector(enabled=True, name='flux-system', namespace='flux-system', skip_crds=True, skip_secrets=True, visitor=None), kustomization=MetadataSelector(enabled=True, name=None, namespace=None, skip_crds=True, skip_secrets=True, visitor=ResourceVisitor(func=<bound method ContentOutput.call_async of <flux_local.tool.visitor.ContentOutput object at 0x1041fd290>>)), helm_repo=MetadataSelector(enabled=True, name=None, namespace=None, skip_crds=True, skip_secrets=True, visitor=ResourceVisitor(func=<function HelmVisitor.repo_visitor.<locals>.add_repo at 0x105215e40>)), helm_release=MetadataSelector(enabled=True, name=None, namespace=None, skip_crds=True, skip_secrets=True, visitor=ResourceVisitor(func=<function HelmVisitor.release_visitor.<locals>.add_release at 0x105216020>)), cluster_policy=MetadataSelector(enabled=True, name=None, namespace=None, skip_crds=True, skip_secrets=True, visitor=None))
DEBUG:git.util:Failed checking if running in CYGWIN due to: FileNotFoundError(2, 'No such file or directory')
DEBUG:git.cmd:Popen(['git', 'rev-parse', '--show-toplevel'], cwd=/private/tmp/flux, universal_newlines=False, shell=None, istream=None)
DEBUG:flux_local.command:Running command: (/private/tmp/flux) kustomize cfg grep kind=Kustomization .
DEBUG:flux_local.command:Running command: (None) kustomize cfg grep 'spec.sourceRef.kind=GitRepository|OCIRepository'
DEBUG:flux_local.git_repo:roots=[Kustomization(name='demo', namespace='flux-system', path='./base', helm_repos=[], helm_releases=[], cluster_policies=[], source_path='kustomization.yaml', source_kind='GitRepository', source_name='flux-system', source_namespace='flux-system')]
DEBUG:flux_local.git_repo:No clusters found; Processing as a Kustomization: .
DEBUG:flux_local.git_repo:Building cluster cluster .
DEBUG:flux_local.git_repo:Visiting path (.) .
DEBUG:flux_local.command:Running command: (/private/tmp/flux) kustomize cfg grep kind=Kustomization .
DEBUG:flux_local.command:Running command: (None) kustomize cfg grep 'spec.sourceRef.kind=GitRepository|OCIRepository'
DEBUG:flux_local.git_repo:Found 1 Kustomizations (1 unique)
DEBUG:flux_local.git_repo:Kustomization 'demo' sourceRef.kind 'GitRepository' of 'flux-system'
DEBUG:flux_local.git_repo:Visiting path (.) base
DEBUG:flux_local.command:Running command: (/private/tmp/flux/base) kustomize cfg grep kind=Kustomization .
DEBUG:flux_local.command:Running command: (None) kustomize cfg grep 'spec.sourceRef.kind=GitRepository|OCIRepository'
DEBUG:flux_local.git_repo:Found 0 Kustomizations (0 unique)
DEBUG:flux_local.git_repo:Processing kustomization 'demo': base
DEBUG:flux_local.command:Running command: (/private/tmp/flux/base) kustomize build --load-restrictor=LoadRestrictionsNone
DEBUG:flux_local.command:Running command: (None) kustomize cfg grep 'kind=^(CustomResourceDefinition|Secret)$' --invert-match
DEBUG:flux_local.command:Running command: (None) kustomize cfg grep 'kind=^(HelmRepository|HelmRelease|ClusterPolicy)$'
DEBUG:flux_local.tool.visitor:Waiting for cluster inflation to complete
DEBUG:flux_local.tool.visitor:Inflating Helm charts in cluster .
DEBUG:flux_local.helm:Updating 1 repositories
DEBUG:flux_local.command:Running command: (None) helm repo update --registry-config /dev/null --repository-cache /var/folders/tj/25qdsj_n1bqfr1bwvj9hjqxw0000gq/T/tmpas9e_86w --repository-config /var/folders/tj/25qdsj_n1bqfr1bwvj9hjqxw0000gq/T/tmprf2kvdxh/repository-config.yaml
DEBUG:flux_local.tool.visitor:Waiting for tasks to inflate .
DEBUG:flux_local.command:Running command: (None) helm template demo flux-system-demo/demo --namespace flux-system --skip-crds --skip-tests --version 0.1.0 --registry-config /dev/null --repository-cache /var/folders/tj/25qdsj_n1bqfr1bwvj9hjqxw0000gq/T/tmpas9e_86w --repository-config /var/folders/tj/25qdsj_n1bqfr1bwvj9hjqxw0000gq/T/tmprf2kvdxh/repository-config.yaml
ERROR:flux_local.command:Command '(None) helm template demo flux-system-demo/demo --namespace flux-system --skip-crds --skip-tests --version 0.1.0 --registry-config /dev/null --repository-cache /var/folders/tj/25qdsj_n1bqfr1bwvj9hjqxw0000gq/T/tmpas9e_86w --repository-config /var/folders/tj/25qdsj_n1bqfr1bwvj9hjqxw0000gq/T/tmprf2kvdxh/repository-config.yaml' failed with return code 1
Error: no cached repo found. (try 'helm repo update'): open /var/folders/tj/25qdsj_n1bqfr1bwvj9hjqxw0000gq/T/tmpas9e_86w/flux-system-demo-index.yaml: no such file or directory

Traceback (most recent call last):
  File "/opt/homebrew/lib/python3.11/site-packages/flux_local/tool/flux_local.py", line 60, in main
    asyncio.run(action.run(**vars(args)))
  File "/opt/homebrew/Cellar/python@3.11/3.11.4_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/runners.py", line 190, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.11/3.11.4_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.11/3.11.4_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/base_events.py", line 653, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/flux_local/tool/build.py", line 94, in run
    await helm_visitor.inflate(
  File "/opt/homebrew/lib/python3.11/site-packages/flux_local/tool/visitor.py", line 296, in inflate
    await asyncio.gather(*tasks)
  File "/opt/homebrew/lib/python3.11/site-packages/flux_local/tool/visitor.py", line 324, in inflate_cluster
    await asyncio.gather(*tasks)
  File "/opt/homebrew/lib/python3.11/site-packages/flux_local/tool/visitor.py", line 221, in inflate_release
    await visitor.func(cluster_path, pathlib.Path(""), release, cmd)
  File "/opt/homebrew/lib/python3.11/site-packages/flux_local/tool/visitor.py", line 135, in call_async
    content = await cmd.run()
              ^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/flux_local/kustomize.py", line 114, in run
    return await run_piped(self._cmds)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/flux_local/command.py", line 104, in run_piped
    result = await _run_piped_with_sem(cmds)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/flux_local/command.py", line 96, in _run_piped_with_sem
    out = await asyncio.wait_for(cmd.run(stdin), _TIMEOUT)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.11/3.11.4_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/tasks.py", line 479, in wait_for
    return fut.result()
           ^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/flux_local/command.py", line 87, in run
    raise self.exc("\n".join(errors))
flux_local.exceptions.HelmException: Command '(None) helm template demo flux-system-demo/demo --namespace flux-system --skip-crds --skip-tests --version 0.1.0 --registry-config /dev/null --repository-cache /var/folders/tj/25qdsj_n1bqfr1bwvj9hjqxw0000gq/T/tmpas9e_86w --repository-config /var/folders/tj/25qdsj_n1bqfr1bwvj9hjqxw0000gq/T/tmprf2kvdxh/repository-config.yaml' failed with return code 1
Error: no cached repo found. (try 'helm repo update'): open /var/folders/tj/25qdsj_n1bqfr1bwvj9hjqxw0000gq/T/tmpas9e_86w/flux-system-demo-index.yaml: no such file or directory

flux-local error:  Command '(None) helm template demo flux-system-demo/demo --namespace flux-system --skip-crds --skip-tests --version 0.1.0 --registry-config /dev/null --repository-cache /var/folders/tj/25qdsj_n1bqfr1bwvj9hjqxw0000gq/T/tmpas9e_86w --repository-config /var/folders/tj/25qdsj_n1bqfr1bwvj9hjqxw0000gq/T/tmprf2kvdxh/repository-config.yaml' failed with return code 1
Error: no cached repo found. (try 'helm repo update'): open /var/folders/tj/25qdsj_n1bqfr1bwvj9hjqxw0000gq/T/tmpas9e_86w/flux-system-demo-index.yaml: no such file or directory

You can reproduce the same behavior with helm when adding the repo without a password:

$ helm repo add myrepo https://gitlab.example.com/api/v4/projects/1234/packages/helm/stable

Error: looks like "https://gitlab.example.com/api/v4/projects/1234/packages/helm/stable" is not a valid chart repository or cannot be reached: failed to fetch https://gitlab.example.com/api/v4/projects/1234/packages/helm/stable/index.yaml : 401 Unauthorized

The same command succeeds with username and password:

$ helm repo add myrepo https://gitlab.example.com/api/v4/projects/1234/packages/helm/stable --username token --password <token> 

"myrepo" has been added to your repositories

Interestingly, helm already fails when adding the repository. flux-local only terminates with an error when I also add a helm release.

@allenporter is there something I can further assist with?

Thanks for the extra detail -- sorry this fell off my radar. The way this works now in flux local is that instead of adding the repo one at a time, it makes the repository config file:

flux-local/flux_local/helm.py

Line 75 in b15d571

def config(self) -> dict[str, Any]:

then it runs the help template command

flux-local/flux_local/helm.py

Line 205 in b15d571

args: list[str] = [

to make it all work. I think we need to either see if this can be done in the config file or in the helm template command. I see helm template has a --password and a --username flag so that looks like it can be used.

I think the steps we need are:

when parsing the HelmRepository track that it uses a secret here

flux-local/flux_local/manifest.py

Line 180 in b15d571

class HelmRepository(BaseManifest):

as fields
when creating the helm template command pass the secret information as flags object here

flux-local/flux_local/helm.py

Line 205 in b15d571

args: list[str] = [

I realize though we need a way to pass a secret for that repository... Maybe another command line flag similar to --sources, or maybe an enviroment variable, or maybe it can find it from another object in the cluster (e.g. you create a fake Secret yaml file while running under test. There is prior art for something like this in real CI systems... Curious if you have an opinion or know best practices here.

@allenporter Thanks for the explanation of what happens in the code. I'm going to take some time to get my head around the options here.
But before I found out that instead of

---
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: HelmRepository
[...]
  url: https://gitlab.example.com/api/v4/projects/1234/packages/helm/stable
  secretRef:
    name: helm-pull-token

I can also pass the token via the URL and postBuild variable substitution like this

---
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: HelmRepository
[...]
  url: https://token:<token>@gitlab.example.com/api/v4/projects/1234/packages/helm/stable

However, I do not think variable substitution is supported by flux-local atm, but perhaps this is an easier approach to implement.

I haven't come up with a perfect solution yet, but I've managed to get it to work with variables substitution before running flux-local.
First of all, I think the easiest way would be to use private OCI charts instead of helm chart artefacts, as you could simply run helm registry login before running flux-local. Unfortunately there is no such thing as helm chart login.

I'm still not sure how we should pass credentials to flux-local, I think the best way would be to pass username and password to the helm repo add command based on the secretRef in the HelmRelease resource. This would require inventing some sort of "virtual secret" to pass these credentials.

As for the environment variable part, I'm not sure about the naming of the variables. In my case, only one token is needed for each chart, but I think a solution should support different tokens per helm repository. This would require environment variables like HELM_REPO_<helm-repo-name>_USERNAME and HELM_REPO_<helm-repo-name>_PASSWORD.
In this case, we could check for a matching set of environment variables for the current helm repository and pass the --username and --password parameters to the helm command.

Maybe we can add a flag where secretes are passed in as key/value pairs, then referenced as needed.

Hello all,
Was wondering if there was an update on this issue in general, or maybe there is another way to skip specific helmreleases/apps that require helm registry login with something like --ignore ?

@xakaitetoia
We'll still use the workaround I briefly mentioned in my last post.
We configure the HelmRepositories as follows:

---
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: HelmRepository
metadata:
  name: myhelmrepo
  namespace: flux-system
spec:
  interval: 1m
  url: https://token:${helm_pull_token}@gitlab.example.com/api/v4/projects/1234/packages/helm/stable

We run a script in our pipeline that recursively replaces the helm_pull_token in all files in a given subdirectory with an actual token created at runtime:

#!/bin/bash

folder_path="$1"

replace_variable_in_file() {
    file_path="$1"
    envsubst '$helm_pull_token' < "$file_path" > temp.txt
    mv temp.txt "$file_path"
}

search_files_recursive() {
    local current_folder="$1"
    for file in "$current_folder"/*; do
        if [ -f "$file" ]; then
            replace_variable_in_file "$file"
        elif [ -d "$file" ]; then
            search_files_recursive "$file"
        fi
    done
}
search_files_recursive "$folder_path"

Not a pretty solution, but it gets the job done.

In #717 it was proposed to add a --repository-config which is now available 5.4.0. Perhaps this will let you specify authentication for a private repository?

I think this should work, although we'll be moving to OCI charts in the near future, so we'll end up with the exact same setup that was fixed in #717. We can close this issue from my point of view.

Thanks!