Feature Request: add Argocd application autodiscovery

Is your feature request related to a problem?

I would like to automatically update the helm charts managed by the Argocd application manifest a repoURL sets the upstream chart and the targetRevision sets the version number.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: sealed-secrets
  namespace: argocd
spec:
  project: default
  source:
    chart: sealed-secrets
    repoURL: https://bitnami-labs.github.io/sealed-secrets
    targetRevision: 1.16.1
    helm:
      releaseName: sealed-secrets
  destination:
    server: "https://kubernetes.default.svc"
    namespace: kubeseal

Solution you'd like

I would like it to look for all files in a repo.

Alternatives you've considered

No response

Anything else?

No response

Hey,
It would definitely makes.
This could be new autodiscovery plugin named argocd

The goal of a autodiscovery is to parse files to gather all information needed to generate Updatecli manifest

The Rancher fleet plugin could probably be used as a source of inspiration

https://github.com/updatecli/updatecli/blob/main/pkg/plugins/autodiscovery/fleet/main.go

Then that new plugin would need to be mapped to Updatecli on the following places

updatecli/pkg/core/pipeline/autodiscovery/main.go

Line 78 in 296e67c

for kind := range g.spec.Crawlers {
updatecli/pkg/core/pipeline/autodiscovery/main.go

Line 40 in 296e67c

AutodiscoverySpecsMapping = map[string]interface{}{
updatecli/pkg/core/pipeline/autodiscovery/main.go

Line 24 in 296e67c

DefaultCrawlerSpecs = Config{

Notes that fleet relies on file named fleet.yaml or fleet.yml, in the context of argocd this could be any yaml files with the field apiVersion set to argoproj.io

@olblak I can work on this how can I assign myself?

@olblak I can work on this how can I assign myself?

More than happy to assign it to you.
Feel free to reach out if you need some assistance.
Also we are a few to hang on Matrix to chat if you want to join us

https://app.gitter.im/#/room/#Updatecli_community:gitter.im

@cloudoutloud have you started this feature yet?
I am almost done with flux support and I could implemented the argocd support next while I am in this area of the code :)

@cloudoutloud have you started this feature yet? I am almost done with flux support and I could implemented the argocd support next while I am in this area of the code :)

Otherwise I could pick up something else as there are plenty stuff to improve anyway

@olblak
I briefly looked at what was need for the change.
Planning to look at week commencing 4th March as away on holiday now 😀

Awesome thanks, then I'll pick something else, enjoy your vacation :)

Any updates on this? Integration with ArgoCD or a plugin would be greatly appreciated

I think @cloudoutloud wanted to look into this but I am fine picking up here.

@olblak
Apologies feel free to pick this up, I have not found the time.
I don't want to stall the implementation.

Apologies feel free to pick this up, I have not found the time.

No need to apologies, don't worry that's totally fine.
I'll try to find some time to implement the support

I opened an pullrequest and I am looking for a real argocd git repository containing applications to update :)

@cloudoutloud @paltaa

My pullrequest only handles helm version update.
Feel free to share if other kind of version could be updated in a argocd manifest

#######################
# LOCAL AUTODISCOVERY #
#######################


###########################################################################################################
# DEPS(HELM): BUMP HELM CHART "SEALED-SECRETS" IN ARGOCD MANIFEST "TESTDATA/SEALED-SECRETS/MANIFEST.YAML" #
###########################################################################################################


SOURCES
=======
WARNING: Updatecli binary version is unset. This means you are using a development version that ignores manifest version constraint.

sealed-secrets
--------------
Searching for version matching pattern "*"
✔ Helm Chart "sealed-secrets" version "2.15.3" is found from repository "https://bitnami-labs.github.io/sealed-secrets"

Remark: We couldn't identify a way to automatically retrieve changelog information.
Please use following information to take informed decision

Helm Chart: sealed-secrets
Helm chart for the sealed-secrets controller.
Project Home: https://github.com/bitnami-labs/sealed-secrets
Require Kubernetes Version: &gt;=1.16.0-0
Version created on the 2024-04-04 16:40:40.720237525 &#43;0000 UTC

Sources:

* https://github.com/bitnami-labs/sealed-secrets



URL:

* https://github.com/bitnami-labs/sealed-secrets/releases/download/helm-v2.15.3/sealed-secrets-2.15.3.tgz




CHANGELOG:
----------

Remark: We couldn't identify a way to automatically retrieve changelog information.
Please use following information to take informed decision

Helm Chart: sealed-secrets
Helm chart for the sealed-secrets controller.
Project Home: https://github.com/bitnami-labs/sealed-secrets
Require Kubernetes Version: &gt;=1.16.0-0
Version created on the 2024-04-04 16:40:40.720237525 &#43;0000 UTC

Sources:

* https://github.com/bitnami-labs/sealed-secrets



URL:

* https://github.com/bitnami-labs/sealed-secrets/releases/download/helm-v2.15.3/sealed-secrets-2.15.3.tgz





CONDITIONS:
===========

sealed-secrets-name
-------------------
✔ key "$.spec.source.chart" is correctly set to "sealed-secrets"

sealed-secrets-repository
-------------------------
✔ key "$.spec.source.repoURL" is correctly set to "https://bitnami-labs.github.io/sealed-secrets"


TARGETS
========

sealed-secrets
--------------

**Dry Run enabled**

⚠ - change detected:
	* key "$.spec.source.targetRevision" should be updated from "1.16.1" to "2.15.3", in file "testdata/sealed-secrets/manifest.yaml"


ACTIONS
========


=============================

REPORTS:



✔ Local AutoDiscovery:

⚠ deps(helm): bump Helm chart "sealed-secrets" in ArgoCD manifest "testdata/sealed-secrets/manifest.yaml":
	Source:
		✔ [sealed-secrets] Get latest "sealed-secrets" Helm chart version
	Condition:
		✔ [sealed-secrets-name] Ensure Helm chart name sealed-secrets is specified
		✔ [sealed-secrets-repository] Ensure Helm chart repository https://bitnami-labs.github.io/sealed-secrets is specified
	Target:
		⚠ [sealed-secrets] deps(helm): bump Helm chart "sealed-secrets" in ArgoCD manifest "testdata/sealed-secrets/manifest.yaml"


Run Summary
===========
Pipeline(s) run:
  * Changed:	1
  * Failed:	0
  * Skipped:	0
  * Succeeded:	1
  * Total:	2

Hey @olblak, is there a possibility to specify the keys to upgrade? right now im using multiple sources for some of the apps, also if you got any instructions to test your PR i would be happy to do so,

Cheers

if you got any instructions to test your PR i would be happy to do so,

@paltaa you can test the pullrequest by running

# Build a new Updatecli binary from the root directory of my PR
go build -o bin/updatecli .
# Execute Updatecli from the root directory of you argocd project
./bin/updatecli diff
# To see generated manifest
./bin/updatecli manifest show

is there a possibility to specify the keys to upgrade

do you have more keys than targetRevision to update?

@paltaa I just released argocd support in https://github.com/updatecli/updatecli/releases/tag/v0.77.0 with docs on https://www.updatecli.io/docs/plugins/autodiscovery/argocd/

Hey @olblak i've been testing the plugin with autodiscovery but the PRs are not being created and im not sure why, cant see anything that could point me to the right direction in the logs with debug:

➜  gitops git:(main) ✗ updatecli diff --debug --values updatecli.d/cfg.yaml


+++++++++++
+ PREPARE +
+++++++++++

DEBUG: Default Updatecli manifest directory detected "updatecli.d"
Loading Pipeline "updatecli.d/cfg.yaml"
DEBUG: pipelineid undefined, we'll try to generate one
DEBUG: pipeline name undefined, we'll use the manifest file checksum
Loading Pipeline "updatecli.d/default.yaml"
DEBUG: pipelineid undefined, we'll try to generate one
DEBUG: using pipeline name to generate the pipelineid

SCM repository retrieved: 1
DEBUG: cloning git repository: https://github.com/Harbor-Systems/gitops.git in /var/folders/kp/shczqhd56kx8td73p3gzl5j40000gn/T/updatecli/github/Harbor-Systems/gitops
DEBUG: cloning git repository: https://github.com/Harbor-Systems/gitops.git in /var/folders/kp/shczqhd56kx8td73p3gzl5j40000gn/T/updatecli/github/Harbor-Systems/gitops
DEBUG: repository already exists, trying to pull changes
DEBUG: Fetching remote branches for resetting local ones


++++++++++++++++++
+ AUTO DISCOVERY +
++++++++++++++++++



++++++++++++++++++++++++++++++++
+ ARGOCD AUTODISCOVERY EXAMPLE +
++++++++++++++++++++++++++++++++

DEBUG: checkout git branch "updatecli_main_26212ee07681e41117e1615c809e501e3b20d6236dc8113ae90444c103f0f84e", based on "main"
DEBUG: Checking if branch "updatecli_main_26212ee07681e41117e1615c809e501e3b20d6236dc8113ae90444c103f0f84e" diverged from "main":
DEBUG: 	all good,branch "updatecli_main_26212ee07681e41117e1615c809e501e3b20d6236dc8113ae90444c103f0f84e" is ahead of "main"


ARGOCD
=======
DEBUG: ArgoCD manifest: "/var/folders/kp/shczqhd56kx8td73p3gzl5j40000gn/T/updatecli/github/Harbor-Systems/gitops/helm/dev/prometheus-operator-crds/application.yaml"
DEBUG: Helm Chart Name: "prometheus-operator-crds"
DEBUG: Helm Repository URL: "https://prometheus-community.github.io/helm-charts"
DEBUG: Version: "7.0.0"
DEBUG: ArgoCD manifest: "/var/folders/kp/shczqhd56kx8td73p3gzl5j40000gn/T/updatecli/github/Harbor-Systems/gitops/helm/prod/prometheus-operator-crds/application.yaml"
DEBUG: Helm Chart Name: "prometheus-operator-crds"
DEBUG: Helm Repository URL: "https://prometheus-community.github.io/helm-charts"
DEBUG: Version: "7.0.0"
DEBUG: 2 Argocd manifest(s) found
DEBUG:     * "helm/dev/prometheus-operator-crds/application.yaml"
DEBUG:     * "helm/prod/prometheus-operator-crds/application.yaml"
DEBUG: parsing file "/var/folders/kp/shczqhd56kx8td73p3gzl5j40000gn/T/updatecli/github/Harbor-Systems/gitops/helm/dev/prometheus-operator-crds/application.yaml"
DEBUG: ArgoCD manifest: "/var/folders/kp/shczqhd56kx8td73p3gzl5j40000gn/T/updatecli/github/Harbor-Systems/gitops/helm/dev/prometheus-operator-crds/application.yaml"
DEBUG: Helm Chart Name: "prometheus-operator-crds"
DEBUG: Helm Repository URL: "https://prometheus-community.github.io/helm-charts"
DEBUG: Version: "7.0.0"
DEBUG: file path "helm/dev/prometheus-operator-crds/application.yaml" matching rule "helm/dev/*/application.yaml"
DEBUG: parsing file "/var/folders/kp/shczqhd56kx8td73p3gzl5j40000gn/T/updatecli/github/Harbor-Systems/gitops/helm/prod/prometheus-operator-crds/application.yaml"
DEBUG: ArgoCD manifest: "/var/folders/kp/shczqhd56kx8td73p3gzl5j40000gn/T/updatecli/github/Harbor-Systems/gitops/helm/prod/prometheus-operator-crds/application.yaml"
DEBUG: Helm Chart Name: "prometheus-operator-crds"
DEBUG: Helm Repository URL: "https://prometheus-community.github.io/helm-charts"
DEBUG: Version: "7.0.0"
DEBUG: Ignoring Helm chart "prometheus-operator-crds" from "helm/prod/prometheus-operator-crds/application.yaml", as not matching only rule(s)
Manifest detected: 1


---

=> Total manifest detected: 1



++++++++++++
+ PIPELINE +
++++++++++++



############
# CFG.YAML #
############


################################
# ARGOCD AUTODISCOVERY EXAMPLE #
################################


##################################################################################################################################
# DEPS(HELM): BUMP HELM CHART "PROMETHEUS-OPERATOR-CRDS" IN ARGOCD MANIFEST "HELM/DEV/PROMETHEUS-OPERATOR-CRDS/APPLICATION.YAML" #
##################################################################################################################################


SOURCES
=======

prometheus-operator-crds
------------------------
Searching for version matching pattern "*"
✔ Helm Chart "prometheus-operator-crds" version "11.0.0" is found from repository "https://prometheus-community.github.io/helm-charts"

Remark: We couldn't identify a way to automatically retrieve changelog information.
Please use following information to take informed decision

Helm Chart: prometheus-operator-crds
A Helm chart that collects custom resource definitions (CRDs) from the Prometheus Operator, allowing for seamless integration with GitOps tools

Require Kubernetes Version: &gt;=1.16.0-0
Version created on the 2024-04-06 20:25:10.906667449 &#43;0000 UTC

Sources:

* https://github.com/prometheus-community/helm-charts



URL:

* https://github.com/prometheus-community/helm-charts/releases/download/prometheus-operator-crds-11.0.0/prometheus-operator-crds-11.0.0.tgz




CHANGELOG:
----------

Remark: We couldn't identify a way to automatically retrieve changelog information.
Please use following information to take informed decision

Helm Chart: prometheus-operator-crds
A Helm chart that collects custom resource definitions (CRDs) from the Prometheus Operator, allowing for seamless integration with GitOps tools

Require Kubernetes Version: &gt;=1.16.0-0
Version created on the 2024-04-06 20:25:10.906667449 &#43;0000 UTC

Sources:

* https://github.com/prometheus-community/helm-charts



URL:

* https://github.com/prometheus-community/helm-charts/releases/download/prometheus-operator-crds-11.0.0/prometheus-operator-crds-11.0.0.tgz





CONDITIONS:
===========

prometheus-operator-crds-repository
-----------------------------------
DEBUG: checkout git branch "updatecli_main_26212ee07681e41117e1615c809e501e3b20d6236dc8113ae90444c103f0f84e", based on "main"
DEBUG: Checking if branch "updatecli_main_26212ee07681e41117e1615c809e501e3b20d6236dc8113ae90444c103f0f84e" diverged from "main":
DEBUG: 	all good,branch "updatecli_main_26212ee07681e41117e1615c809e501e3b20d6236dc8113ae90444c103f0f84e" is ahead of "main"
DEBUG: Relative path detected: changing from "helm/dev/prometheus-operator-crds/application.yaml" to absolute path from SCM: "/var/folders/kp/shczqhd56kx8td73p3gzl5j40000gn/T/updatecli/github/Harbor-Systems/gitops/helm/dev/prometheus-operator-crds/application.yaml"
✔ key "$.spec.source.repoURL" is correctly set to "https://prometheus-community.github.io/helm-charts"

prometheus-operator-crds-name
-----------------------------
DEBUG: checkout git branch "updatecli_main_26212ee07681e41117e1615c809e501e3b20d6236dc8113ae90444c103f0f84e", based on "main"
DEBUG: Checking if branch "updatecli_main_26212ee07681e41117e1615c809e501e3b20d6236dc8113ae90444c103f0f84e" diverged from "main":
DEBUG: 	all good,branch "updatecli_main_26212ee07681e41117e1615c809e501e3b20d6236dc8113ae90444c103f0f84e" is ahead of "main"
DEBUG: Relative path detected: changing from "helm/dev/prometheus-operator-crds/application.yaml" to absolute path from SCM: "/var/folders/kp/shczqhd56kx8td73p3gzl5j40000gn/T/updatecli/github/Harbor-Systems/gitops/helm/dev/prometheus-operator-crds/application.yaml"
✔ key "$.spec.source.chart" is correctly set to "prometheus-operator-crds"


TARGETS
========

prometheus-operator-crds
------------------------

**Dry Run enabled**

DEBUG: checkout git branch "updatecli_main_26212ee07681e41117e1615c809e501e3b20d6236dc8113ae90444c103f0f84e", based on "main"
DEBUG: Checking if branch "updatecli_main_26212ee07681e41117e1615c809e501e3b20d6236dc8113ae90444c103f0f84e" diverged from "main":
DEBUG: 	all good,branch "updatecli_main_26212ee07681e41117e1615c809e501e3b20d6236dc8113ae90444c103f0f84e" is ahead of "main"
DEBUG: Relative path detected: changing from "helm/dev/prometheus-operator-crds/application.yaml" to absolute path from SCM: "/var/folders/kp/shczqhd56kx8td73p3gzl5j40000gn/T/updatecli/github/Harbor-Systems/gitops/helm/dev/prometheus-operator-crds/application.yaml"
⚠ - change detected:
	* key "$.spec.source.targetRevision" should be updated from "'7.0.0'" to "11.0.0", in file "helm/dev/prometheus-operator-crds/application.yaml"
DEBUG: Checking if local changes have been done that should be published
DEBUG: no changes detected between branches "main" and "updatecli_main_26212ee07681e41117e1615c809e501e3b20d6236dc8113ae90444c103f0f84e"


ACTIONS
========


=============================

REPORTS:



✔ Argocd Autodiscovery Example:

- CFG.YAML:

⚠ deps(helm): bump Helm chart "prometheus-operator-crds" in ArgoCD manifest "helm/dev/prometheus-operator-crds/application.yaml":
	Source:
		✔ [prometheus-operator-crds] Get latest "prometheus-operator-crds" Helm chart version
	Condition:
		✔ [prometheus-operator-crds-name] Ensure Helm chart name prometheus-operator-crds is specified
		✔ [prometheus-operator-crds-repository] Ensure Helm chart repository https://prometheus-community.github.io/helm-charts is specified
	Target:
		⚠ [prometheus-operator-crds] deps(helm): bump Helm chart "prometheus-operator-crds" in ArgoCD manifest "helm/dev/prometheus-operator-crds/application.yaml"


Run Summary
===========
Pipeline(s) run:
  * Changed:	1
  * Failed:	0
  * Skipped:	1
  * Succeeded:	1
  * Total:	3

Also what i ment by specifying the keys to update is that in ArgoCD applications you can use multiple sources for the same application, for example (just the sources):

  sources:
    - chart: emqx
      repoURL: 'https://repos.emqx.io/charts'
      targetRevision: '5.5.1'
      helm:
        valueFiles:
          - $values/helm/dev/emqx/values.yaml
    - repoURL: 'redacted
      targetRevision: HEAD
      ref: values

In this case i would like to update targetRevision only for chart: emqx. This is the way its setup so you can get values from another or the same repository using gitops for external charts.

Is this supported? Any hints on why the PRs are not created?

Cheers

Mmmh I wasn't aware of sources, I just implemented source as described in this issue
I am reopening this issue

cfr https://argo-cd.readthedocs.io/en/stable/user-guide/multiple_sources/#helm-value-files-from-external-git-repository

@olblak Thanks! Let me know when you have something to test

@paltaa Another pullrequest that you can test.
All you need is building a new binary running go build -o bin/updatecli . and then use it on your repo

@olblak Perfect! Thanks man, will test it right now and comment back

Alright!

Run Summary
===========
Pipeline(s) run:
  * Changed:	1
  * Failed:	3
  * Skipped:	27
  * Succeeded:	23
  * Total:	54

Charts that failed are hosted in private repositories so this should be expected, thanks a lot man!

The only doubt that i have now is why the PRs are not being created? am i missing any config? this is the config i have right now:

updatecli.d/default.yaml

name: Argocd Autodiscovery Example

autodiscovery:
  # scmid is applied to all crawlers
  scmid: default
  crawlers:
    argocd:
      # To ignore specific path
      ignore:
        # Ignore manifest match path "dev/*.yaml"
        - path: helm/prod/*.yaml
        # Ignore any Helm chart named "kubewarden-crds"
        - charts:
            "kubewarden-crds": ""
        # Ignore any helm chart repository named "https://charts.jetstack.io"
        - repositories:
            - "https://charts.jetstack.io"

        # # Ignore any Helm chart named "kubewarden-crds" in manifest path "dev/*.yaml"
        # - path: dev/*.yaml
        #   charts:
        #     "kubewarden-crds": ""
      # Only accepts the same rule than ignore
      only:
       - path: helm/dev/*/application.yaml
      #  - charts:
      #      "kubewarden-crds": ""
      #  - repositories:
      #      - "https://charts.jetstack.io"
      
scms:
  default:
    kind: github
    spec:
      #user: redacted
      owner: redacted
      username: redacted
      token: '{{ requiredEnv "GITHUB_TOKEN" }}'
      repository: gitops
      branch: "main"
    
actions:
    default:
        kind: "github/pullrequest"
        spec:
            labels:
                - "dependencies"
            mergemethod: "squash"
        scmid: "default"

Also, if i run an apply, the only changes commited to the new branch just have 1 source, only one case for our gitops repo

The only doubt that i have now is why the PRs are not being created?

You are missing the actionid in the autodiscovery

autodiscovery:
  # scmid is applied to all crawlers
  scmid: default

should become

autodiscovery:
  actionid: default
  # scmid is applied to all crawlers
  scmid: default

Also, if i run an apply, the only changes commited to the new branch just have 1 source, only one case for our gitops repo

I am wondering why you have some many skipped pipelines according your run summary

Run Summary
===========
Pipeline(s) run:
  * Changed:	1
  * Failed:	3
  * Skipped:	27
  * Succeeded:	23
  * Total:	54

Hey @olblak, added your change and the PR is created but still just commits the change for my one chart with one source, any other config needed?

Ok, it does not like repoURLs with ' '. After removing them the commits are being done! so thanks man.
✗ key "$.spec.sources[0].repoURL", is incorrectly set to "'https://victoriametrics.github.io/helm-charts/'" and should be "https://victoriametrics.github.io/helm-charts/"

Also, all of the skipped is the production application that i dont want to automatically update, so that is working fine

EDIT: Everything works now

Just one last thing, only if possible, can we have 1 pr per dependency? instead of 1 pr for all the upgrades? hehe

@paltaa

Ok, it does not like repoURLs with ' '. After removing them the commits are being done! so thanks man.

Do you think it it's an issue on Updatecli? Was your manifest initially working? Because I would imagine argocd to fail as well

Just one last thing, only if possible, can we have 1 pr per dependency? instead of 1 pr for all the upgrades? hehe

Yes can set groupby to either individual or all where individual opens one pull request per dependency while all group them all like here

Also combined with versionfilter you could group all patch update together
I realise now the indentation for versionfilter documentation is broken on https://www.updatecli.io/docs/plugins/autodiscovery/argocd/

Great! Thanks for all the help man, really appreciate it.

'' were working with ArgoCD so i would think its just an updatecli issue

Hey @olblak any date for the release which will contain this new changes?

Hey @olblak any date for the release which will contain this new changes?

What about now?
Give a moment