The "poll-helm-hub" and "url" features in the config file no longer work
TACY-octo opened this issue · 15 comments
What happened?
Hello, After the Nova 3.0.0 update, the "url:" and "poll-helm-hub:" features are no longer present.
As a result, I find myself with versions of charts in nightly-build and not in release version or tags that no longer corresponds to the original convention : example for "kubed".
Also, Artifacthub does not have the set of available helms : example for "csi-secrets-store".
Example output (2.3.0) :
Release Name Installed Latest Old Deprecated
vector-aggregator 0.18.1 0.21.2 true false
csi-secrets-store 0.1.0 1.0.1 true false
kubed v0.12.0 v0.13.1 true false
And
Example output (3.0.0) :
Release Name Installed Latest Old Deprecated
============ ========= ====== === ==========
kubed v0.12.0 0.13.1 true false
vector-aggregator 0.18.1 0.20.0-nightly-2022-01-04 true false
In 3.0.0 output, we don't have the csi-secrets-store, the vector-aggregator have a nightly build verison and kubed change the tag convention
What did you expect to happen?
After run the command `nova find --config=config.ymlà, I expect to have a result that corresponds to a check between the urls charts of my config file and the currently installed charts in my cluster K8S.
How can we reproduce this?
Use the config file :
#config.yml
poll-helm-hub: false
url:
- https://charts.appscode.com/stable
- https://helm.vector.dev
- https://raw.githubusercontent.com/Azure/secrets-store-csi-driver-provider-azure/master/charts
And run this command on nova 3.0.0 :
nova find --config=config.yml
Result :
Release Name Installed Latest Old Deprecated
============ ========= ====== === ==========
kubed v0.12.0 0.13.1 true false
vector-aggregator 0.18.1 0.20.0-nightly-2022-01-04 true false
And with the 2.3.0 version :
Release Name Installed Latest Old Deprecated
kubed v0.12.0 v0.13.1 true false
csi-secrets-store 0.1.0 1.0.1 true false
vector-aggregator 0.18.1 0.21.2 true false
version
3.0.0
Search
- I did search for other open and closed issues before opening this.
Code of Conduct
- I agree to follow this project's Code of Conduct
Additional context
No response
I would like to add that ArtifactHub has older versions of some charts. For example, kubernetes-replicator has version 1.0.15 on the ArtifactHub, and in the GitHub repo, it is 2.7.3 - link to the Charts.yaml
👋 Hey thanks for this report and apologies for this oversight. We'll use this issue to track a fix on adding back this functionality to bypass using artifacthub.
@ctavian and @klis thanks again for your input here. I've opened PR #74 if you would want to test those out. If you have a working go install on your workstation you can simply pull down this repo + that branch and run make build and then use ./nova like you would normally.
One thing of note is that poll-helm-hub is gone for good but is now replaced by poll-artifacthub. Please comment on that PR if the functionality still doesn't fit your usecases.
@lucasreed, Thanks, it works! :)
@lucasreed & @ctavian are you sure about the logic?
I tried it and looks like that when I specify the URL, only that URL is shown in the output. For example, if I run the default options I get a list of "all" charts that are available on ArtifactHub. Pay attention to the kubernetes-replicator versions.
> ./nova find
Release Name Installed Latest Old Deprecated
============ ========= ====== === ==========
cert-manager v1.7.1 1.7.1 false false
harbor 1.7.1 1.8.1 true false
operator 0.6.0 0.6.0 false false
kubernetes-replicator 2.7.3 1.0.15 false false
minio-operator 4.3.7 4.3.7 false false
kube-prometheus-stack 18.1.0 32.2.1 true false
loki 2.6.1 2.6.1 false false
prometheus-adapter 2.17.0 3.0.2 true false
traefik 10.14.1 10.14.1 false false
Now, I do the same thing, but only I add 2 flags, one to still poll ArtifactHub, another one to poll kubernetes-replicator chart.
I only got results for the kubernetes-replicator chart. I would love to get all the results, only that charts specified in the --url have precedence over the ArtifactHub.
> ./nova find --url=https://helm.mittwald.de --poll-artifacthub=true
Release Name Installed Latest Old Deprecated
============ ========= ====== === ==========
kubernetes-replicator 2.7.3 2.7.3 false false
And yes, I know that I can specify all the URLs, but if I plug nova in the CI system, then I need to maintain a config file, or the full command for nova find
Hello @klis,
I think there might be a problem with the precedence of the value when you use both --url and the poll-artifacthub to true.
In your case, you just want to add new, more recent repositories, see missing from artifacthub.
However, how will nova behave to choose the version between what is provided by the URL and by artifacthub.
So, what I would do is that if the chart is provided via URL then just skip artifacthub. This still means that if the provided URL has an older version then the artifacthub nova will show the old version (kubernetes-replicator example from my previous comment).
If I add --include-all flag, nova will list all the Helm charts, but only one will have Latest filled.
Would be nice to have all the data, url and artifacthub? Basically, if the Helm chart that is installed is defined in the URL, then go to that URL, and for that chart skip artifacthub.
> ./nova find --url=https://helm.mittwald.de --poll-artifacthub=true --include-all
Release Name Installed Latest Old Deprecated
============ ========= ====== === ==========
cert-manager v1.7.1 false false
harbor 1.7.1 false false
operator 0.6.0 false false
kubernetes-replicator 2.7.3 2.7.3 false false
minio-operator 4.3.7 false false
kube-prometheus-stack 18.1.0 false false
loki 2.6.1 false false
prometheus-adapter 2.17.0 false false
traefik 10.14.1 false false
Maybe I'm going too far, but I would like to see nova do that. I don't know if older versions of nova worked that way since I only recently started to integrate this into our CI workflow.
In my testing it should work with both --url and artifacthub in one run but any charts found at the specified URL's will override artifacthub if it finds the same name chart at the url repository. Could you add the --v=8 to your command so we have more logging?
In my testing it should work with both
--urland artifacthub in one run but any charts found at the specified URL's will override artifacthub if it finds the same name chart at the url repository. Could you add the--v=8to your command so we have more logging?
Sure, here is the full output
> ./nova find --url=https://helm.mittwald.de --poll-artifacthub=true --include-all -v=8
I0217 17:18:53.470284 2113 root.go:99] config not set, using flags only
I0217 17:18:53.473523 2113 root.go:183] Settings: map[context: desired-versions:map[] include-all:true output-file: poll-artifacthub:true url:[https://helm.mittwald.de] wide:false]
I0217 17:18:53.473580 2113 root.go:184] All Keys: [desired-versions url poll-artifacthub context wide include-all output-file]
I0217 17:18:55.576877 2113 artifacthub.go:200] found 3 packages matching 'prometheus-adapter'
I0217 17:18:55.577308 2113 artifacthub.go:200] found 6 packages matching 'kube-prometheus-stack'
I0217 17:18:55.582243 2113 artifacthub.go:200] found 3 packages matching 'jenkins-operator'
I0217 17:18:55.582862 2113 artifacthub.go:200] found 1 packages matching 'kubernetes-replicator'
I0217 17:18:55.583053 2113 artifacthub.go:200] found 4 packages matching 'loki-stack'
I0217 17:18:55.584123 2113 artifacthub.go:200] found 2 packages matching 'minio-operator'
I0217 17:18:55.590694 2113 artifacthub.go:200] found 10 packages matching 'harbor'
I0217 17:18:55.606754 2113 artifacthub.go:200] found 30 packages matching 'traefik'
I0217 17:18:55.694735 2113 artifacthub.go:200] found 54 packages matching 'cert-manager'
I0217 17:18:56.229808 2113 root.go:219] found 113 possible package matches
I0217 17:18:56.230065 2113 chartrepo.go:57] loading 1 chart repositories
I0217 17:18:56.230089 2113 chartrepo.go:60] loading chart repository: https://helm.mittwald.de
I0217 17:18:56.435977 2113 chartrepo.go:173] Got 9 installed releases in the cluster
I0217 17:18:56.436269 2113 output.go:114] deduplicating releases for output
I0217 17:18:56.436292 2113 output.go:121] found duplicate release: 'cert-manager', chart: 'cert-manager', namespace: 'cert-manager'
I0217 17:18:56.436295 2113 output.go:121] found duplicate release: 'harbor', chart: 'harbor', namespace: 'harbor-system'
I0217 17:18:56.436297 2113 output.go:121] found duplicate release: 'operator', chart: 'jenkins-operator', namespace: 'jenkins-system'
I0217 17:18:56.436299 2113 output.go:121] found duplicate release: 'kubernetes-replicator', chart: 'kubernetes-replicator', namespace: 'kubernetes-replicator'
I0217 17:18:56.436301 2113 output.go:121] found duplicate release: 'minio-operator', chart: 'minio-operator', namespace: 'minio-system'
I0217 17:18:56.436303 2113 output.go:121] found duplicate release: 'kube-prometheus-stack', chart: 'kube-prometheus-stack', namespace: 'monitoring-system'
I0217 17:18:56.436305 2113 output.go:121] found duplicate release: 'loki', chart: 'loki-stack', namespace: 'monitoring-system'
I0217 17:18:56.436307 2113 output.go:121] found duplicate release: 'prometheus-adapter', chart: 'prometheus-adapter', namespace: 'monitoring-system'
I0217 17:18:56.436308 2113 output.go:121] found duplicate release: 'traefik', chart: 'traefik', namespace: 'traefik'
Release Name Installed Latest Old Deprecated
============ ========= ====== === ==========
cert-manager v1.7.1 false false
harbor 1.7.1 false false
operator 0.6.0 false false
kubernetes-replicator 2.7.3 2.7.3 false false
minio-operator 4.3.7 false false
kube-prometheus-stack 18.1.0 false false
loki 2.6.1 false false
prometheus-adapter 2.17.0 false false
traefik 10.14.1 false false
@klis can you try pulling and building after my most recent commit. It was a simple one-line commit that allows a function to return a nil instead of an empty struct. I think that fixes this problem.
I reproduced your issue locally and after my change it is working as expected.
Before:
> ./nova find --url=https://helm.mittwald.de --poll-artifacthub=true --include-all
Release Name Installed Latest Old Deprecated
============ ========= ====== === ==========
cert-manager v1.7.1 false false
harbor 1.7.1 false false
kubernetes-replicator 2.7.3 2.7.3 false false
After:
> ./nova find --url=https://helm.mittwald.de --poll-artifacthub=true --include-all
Release Name Installed Latest Old Deprecated
============ ========= ====== === ==========
cert-manager v1.7.1 1.7.1 false false
harbor 1.7.1 1.8.1 true false
kubernetes-replicator 2.7.3 2.7.3 false false
@klis can you try pulling and building after my most recent commit. It was a simple one-line commit that allows a function to return a nil instead of an empty struct. I think that fixes this problem.
I reproduced your issue locally and after my change it is working as expected.
Before:
> ./nova find --url=https://helm.mittwald.de --poll-artifacthub=true --include-all Release Name Installed Latest Old Deprecated ============ ========= ====== === ========== cert-manager v1.7.1 false false harbor 1.7.1 false false kubernetes-replicator 2.7.3 2.7.3 false falseAfter:
> ./nova find --url=https://helm.mittwald.de --poll-artifacthub=true --include-all Release Name Installed Latest Old Deprecated ============ ========= ====== === ========== cert-manager v1.7.1 1.7.1 false false harbor 1.7.1 1.8.1 true false kubernetes-replicator 2.7.3 2.7.3 false false
It was working, but it looks like artifacthub has some issue currently, so I will try again later. With the CLI flags and with the config file.
@lucasreed it is working. But, it looks like that --include-all is broken. So when I set --poll-artifacthub=false and --include-all it should return only version for the URL and list all other charts with Latest column that is empty? But I'm only getting the result for the chart specified in the --url.
> ./nova find --url=https://helm.mittwald.de --poll-artifacthub=false --include-all
Release Name Installed Latest Old Deprecated
============ ========= ====== === ==========
kubernetes-replicator 2.7.3 2.7.3 false false
I would expect something like this
Release Name Installed Latest Old Deprecated
============ ========= ====== === ==========
cert-manager v1.7.1 false false
harbor 1.7.1 false false
operator 0.6.0 false false
kubernetes-replicator 2.7.3 2.7.3 false false
minio-operator 4.3.7 false false
kube-prometheus-stack 18.1.0 false false
loki 2.6.1 false false
prometheus-adapter 2.17.0 false false
traefik 10.14.1 false false