Doesn't work with public helm charts due to request throttling from artifacthub

Question

Doesn't work with public helm charts due to request throttling from artifacthub

Closed this issue 2 years ago · 3 comments

What happened?

nova find only works the first time you run it.

After that, artifacthub immediately throttles you with 429s due to the huge volume of requests nova makes to do its work.

Example:

> nova find -v 5
I1016 13:18:22.086983    7541 root.go:129] config not set, using flags only
I1016 13:18:22.087045    7541 root.go:213] Settings: map[containers:false context: desired-versions:map[] include-all:false output-file: poll-artifacthub:true show-errored-containers:false show-non-semver:false show-old:false url:[] wide:false]
I1016 13:18:22.087068    7541 root.go:214] All Keys: [containers context poll-artifacthub show-non-semver desired-versions url output-file include-all show-errored-containers show-old wide]
I1016 13:18:24.176990    7541 artifacthub.go:411] attempt 1 failed to GET https://artifacthub.io/api/v1/packages/search with status code: 429
I1016 13:18:24.177149    7541 artifacthub.go:411] attempt 1 failed to GET https://artifacthub.io/api/v1/packages/search with status code: 429
I1016 13:18:24.177155    7541 artifacthub.go:411] attempt 1 failed to GET https://artifacthub.io/api/v1/packages/search with status code: 429
I1016 13:18:24.177173    7541 artifacthub.go:411] attempt 1 failed to GET https://artifacthub.io/api/v1/packages/search with status code: 429
I1016 13:18:24.178163    7541 artifacthub.go:411] attempt 1 failed to GET https://artifacthub.io/api/v1/packages/search with status code: 429
I1016 13:18:24.178444    7541 artifacthub.go:411] attempt 1 failed to GET https://artifacthub.io/api/v1/packages/search with status code: 429
I1016 13:18:24.178939    7541 artifacthub.go:411] attempt 1 failed to GET https://artifacthub.io/api/v1/packages/search with status code: 429
I1016 13:18:24.179150    7541 artifacthub.go:411] attempt 1 failed to GET https://artifacthub.io/api/v1/packages/search with status code: 429
I1016 13:18:24.179381    7541 artifacthub.go:411] attempt 1 failed to GET https://artifacthub.io/api/v1/packages/search with status code: 429
I1016 13:18:24.179611    7541 artifacthub.go:411] attempt 1 failed to GET https://artifacthub.io/api/v1/packages/search with status code: 429
I1016 13:18:24.179845    7541 artifacthub.go:411] attempt 1 failed to GET https://artifacthub.io/api/v1/packages/search with status code: 429
I1016 13:18:24.180125    7541 artifacthub.go:411] attempt 1 failed to GET https://artifacthub.io/api/v1/packages/search with status code: 429
I1016 13:18:24.180319    7541 artifacthub.go:411] attempt 1 failed to GET https://artifacthub.io/api/v1/packages/search with status code: 429
I1016 13:18:24.182105    7541 artifacthub.go:411] attempt 2 failed to GET https://artifacthub.io/api/v1/packages/search with status code: 429
I1016 13:18:24.182273    7541 artifacthub.go:411] attempt 2 failed to GET https://artifacthub.io/api/v1/packages/search with status code: 429
I1016 13:18:24.182456    7541 artifacthub.go:411] attempt 2 failed to GET https://artifacthub.io/api/v1/packages/search with status code: 429
I1016 13:18:24.183659    7541 artifacthub.go:411] attempt 2 failed to GET https://artifacthub.io/api/v1/packages/search with status code: 429
I1016 13:18:24.183861    7541 artifacthub.go:411] attempt 2 failed to GET https://artifacthub.io/api/v1/packages/search with status code: 429
I1016 13:18:24.184187    7541 artifacthub.go:411] attempt 2 failed to GET https://artifacthub.io/api/v1/packages/search with status code: 429
I1016 13:18:24.184428    7541 artifacthub.go:411] attempt 2 failed to GET https://artifacthub.io/api/v1/packages/search with status code: 429
I1016 13:18:24.184689    7541 artifacthub.go:411] attempt 2 failed to GET https://artifacthub.io/api/v1/packages/search with status code: 429
I1016 13:18:24.184965    7541 artifacthub.go:411] attempt 2 failed to GET https://artifacthub.io/api/v1/packages/search with status code: 429
I1016 13:18:24.185527    7541 artifacthub.go:411] attempt 2 failed to GET https://artifacthub.io/api/v1/packages/search with status code: 429
I1016 13:18:24.185843    7541 artifacthub.go:411] attempt 2 failed to GET https://artifacthub.io/api/v1/packages/search with status code: 429
I1016 13:18:24.186055    7541 artifacthub.go:411] attempt 2 failed to GET https://artifacthub.io/api/v1/packages/search with status code: 429
I1016 13:18:24.198882    7541 artifacthub.go:302] error GETing response for with search term 'ingress-nginx': error code: 429
I1016 13:18:24.198891    7541 artifacthub.go:265] error searching for term error code: 429
I1016 13:18:24.199245    7541 artifacthub.go:411] attempt 5 failed to GET https://artifacthub.io/api/v1/packages/search with status code: 429
I1016 13:18:24.199255    7541 artifacthub.go:302] error GETing response for with search term 'metrics-server': error code: 429
I1016 13:18:24.199261    7541 artifacthub.go:265] error searching for term error code: 429
I1016 13:18:24.200512    7541 artifacthub.go:411] attempt 4 failed to GET https://artifacthub.io/api/v1/packages/search with status code: 429
I1016 13:18:24.200716    7541 artifacthub.go:411] attempt 5 failed to GET https://artifacthub.io/api/v1/packages/search with status code: 429
I1016 13:18:24.200725    7541 artifacthub.go:302] error GETing response for with search term 'redis': error code: 429
I1016 13:18:24.200732    7541 artifacthub.go:265] error searching for term error code: 429
I1016 13:18:24.200888    7541 artifacthub.go:411] attempt 5 failed to GET https://artifacthub.io/api/v1/packages/search with status code: 429
I1016 13:18:24.200898    7541 artifacthub.go:302] error GETing response for with search term 'mongodb': error code: 429
I1016 13:18:24.200904    7541 artifacthub.go:265] error searching for term error code: 429
I1016 13:18:24.201459    7541 artifacthub.go:411] attempt 5 failed to GET https://artifacthub.io/api/v1/packages/search with status code: 429
I1016 13:18:24.201477    7541 artifacthub.go:302] error GETing response for with search term 'elasticsearch': error code: 429
I1016 13:18:24.201490    7541 artifacthub.go:265] error searching for term error code: 429
I1016 13:18:24.201647    7541 artifacthub.go:411] attempt 5 failed to GET https://artifacthub.io/api/v1/packages/search with status code: 429
I1016 13:18:24.201659    7541 artifacthub.go:302] error GETing response for with search term 'postgresql': error code: 429
I1016 13:18:24.201668    7541 artifacthub.go:265] error searching for term error code: 429
I1016 13:18:24.201853    7541 artifacthub.go:411] attempt 5 failed to GET https://artifacthub.io/api/v1/packages/search with status code: 429
I1016 13:18:24.201861    7541 artifacthub.go:302] error GETing response for with search term 'kibana': error code: 429
I1016 13:18:24.201869    7541 artifacthub.go:265] error searching for term error code: 429
I1016 13:18:24.202696    7541 artifacthub.go:411] attempt 5 failed to GET https://artifacthub.io/api/v1/packages/search with status code: 429
I1016 13:18:24.202715    7541 artifacthub.go:302] error GETing response for with search term 'fluent-bit': error code: 429
I1016 13:18:24.202726    7541 artifacthub.go:265] error searching for term error code: 429
I1016 13:18:24.202870    7541 artifacthub.go:411] attempt 5 failed to GET https://artifacthub.io/api/v1/packages/search with status code: 429
I1016 13:18:24.202881    7541 artifacthub.go:302] error GETing response for with search term 'graylog': error code: 429
I1016 13:18:24.202889    7541 artifacthub.go:265] error searching for term error code: 429
I1016 13:18:24.203079    7541 artifacthub.go:411] attempt 5 failed to GET https://artifacthub.io/api/v1/packages/search with status code: 429
I1016 13:18:24.203087    7541 artifacthub.go:302] error GETing response for with search term 'helm-operator': error code: 429
I1016 13:18:24.203093    7541 artifacthub.go:265] error searching for term error code: 429
I1016 13:18:24.203502    7541 artifacthub.go:411] attempt 5 failed to GET https://artifacthub.io/api/v1/packages/search with status code: 429
I1016 13:18:24.203518    7541 artifacthub.go:302] error GETing response for with search term 'cert-manager': error code: 429
I1016 13:18:24.203527    7541 artifacthub.go:265] error searching for term error code: 429
I1016 13:18:24.203780    7541 artifacthub.go:411] attempt 5 failed to GET https://artifacthub.io/api/v1/packages/search with status code: 429
I1016 13:18:24.203790    7541 artifacthub.go:302] error GETing response for with search term 'sealed-secrets': error code: 429
I1016 13:18:24.203796    7541 artifacthub.go:265] error searching for term error code: 429
I1016 13:18:24.204637    7541 artifacthub.go:411] attempt 5 failed to GET https://artifacthub.io/api/v1/packages/search with status code: 429
I1016 13:18:24.204654    7541 artifacthub.go:302] error GETing response for with search term 'flux': error code: 429
I1016 13:18:24.204664    7541 artifacthub.go:265] error searching for term error code: 429

What did you expect to happen?

Nova should work properly when ran 2 or more times.

How can we reproduce this?

Install nova and run it a couple of times.

Version

Version:3.2.0 Commit:2d64fbc53116ebf8c65e8abf1e9c53039d44a32f

Search

I did search for other open and closed issues before opening this.

Code of Conduct

I agree to follow this project's Code of Conduct

Additional context

No response

Answer 1 · 2022-10-24T17:00:34.000Z

We've switched to declaring the actual Helm charts repo of each project. Instead of the Artifact Hub. default ... it works better and is correct. I'll suggest you to do the same thing.

Answer 2 · 2022-11-17T19:45:08.000Z

My hope is that we will be able get helm/helm#11378 merged into helm, which will allow us to stop polling artifacthub so aggressively. Until then, the suggestion from @LarsBingBong is the way I would go.

Answer 3 · 2022-11-18T07:14:36.000Z

@sudermanjr interesting with helm/helm#11378 ... But, it isn't going to alleviate the sore point that on ArtifactHub the project that Nova then would get is not necessarily the official chart but rather someone that forked the official one and the went in another direction with it. For whatever reason. I way rather want to use the official chart.

Thanks