Helm charts repository index.yaml retention policy

Question

Helm charts repository index.yaml retention policy

carrodher opened this issue 2 years ago · 168 comments

As reported in this issue (#8433), in the last few times we are facing some issues with the index.yaml associated with the Bitnami Helm charts repository.

Current situation

After some investigation, it seems the root cause is related to CloudFront reaching some limits due to the volume of traffic when serving the index.yaml.
This index.yaml contains all the Bitnami Helm charts history (around 15300 entries), producing a pretty fat 14MB file. Given the size of the file and the volume of traffic, thousands of terabytes of download traffic per month are being generated.

One of the alternatives considered was the use of compression at CloudFront, in that case, this solution doesn’t work since compression is not used by the Helm client (helm) itself (see helm/helm#8070) so it doesn’t solve the reported issue.

Mitigation

As the first line of action, we will reduce the size of the index.yaml by removing some old versions and keeping all versions for a period of time (6 months).

⚠️ Please note this action is not removing/deleting any Helm chart, packaged tarballs (.tgz) won't be removed, this action is only affecting index.yaml used to list the Helm charts. Previous versions of the index.yaml can be used to install old Helm charts.

Please note Helm charts tarballs (.tgz) won't be removed, this action is only affecting index.yaml.

Result

Applying this approach (#10530), we obtained the following results:

Total chart versions
* Before: 15260
* Removed: 12138
* After: 3122

Producing a reduced 3.5MB index.yaml.

🔧 Workaround for previous versions

The index.yaml is stored in this repository under the index branch, users should be able to use any commit in that branch to add a previous version of the index.yaml.

Manually using helm repo add

$ helm repo add bitnami-full-index https://raw.githubusercontent.com/bitnami/charts/archive-full-index/bitnami
"bitnami-full-index" has been added to your repositories

When used as a dependency in Chart.yaml:

  - name: postgresql
    version: 8.1.0
-   repository: https://charts.bitnami.com/bitnami
+   repository: https://raw.githubusercontent.com/bitnami/charts/archive-full-index/bitnami
    condition: postgresql.enabled

soylentseth commented 2 years ago

Fuck!

👍10
😄1

Answer 1 · 2022-06-02T11:08:22.000Z

Thank you very much

Answer 2 · 2022-06-02T11:48:42.000Z

Hi,
https://charts.bitnami.com/bitnami seems to return 403 AccessDenied error at the moment.
Is this related to this issue?
My guess is that if you work with S3 bucket as CloudFront origin, the new index.yaml got the wrong ACL now (not publicly readable)

Answer 3 · 2022-06-02T13:31:02.000Z

This is TERRIBLY wrong. For example, we released Helm Chart to our users https://airflow.apache.org/docs/helm-chart/stable/index.html with bitnami's Postgres 10.5.3 and all the charts our users are using are useless now if they use embedded Postgres.

How is it possible you do such a breaking change in such a way ? This must be some kind of a joke.

This totally undermines any trust in bitnami as a source of charts - You basically rendered all historical charts released by likely huge number of projects useless.

Answer 4 · 2022-06-02T14:03:09.000Z

As reported in this issue (#8433), in the last few times we are facing some issues with the index.yaml associated with the Bitnami Helm charts repository.

Current situation

After some investigation, it seems the root cause is related to CloudFront reaching some limits due to the volume of traffic when serving the index.yaml. This index.yaml contains all the Bitnami Helm charts history (around 15300 entries), producing a pretty fat 14MB file. Given the size of the file and the volume of traffic, thousands of terabytes of download traffic per month are being generated.

Mitigation

As the first line of action, we will reduce the size of the index.yaml by removing some old versions and keeping all versions for a period of time (6 months).

Please note Helm charts tarballs (.tgz) won't be removed, this action is only affecting index.yaml.

Result

Applying this approach (#10530), we obtained the following results:
Total chart versions
* Before: 15260
* Removed: 12138
* After: 3122
Producing a reduced 3.5MB index.yaml.

Workaround for previous versions

The index.yaml is stored in this repository under the index branch, users should be able to use any commit in that branch to add a previous version of the index.yaml, for example:
$ helm repo add bitnami-pre-2022 https://raw.githubusercontent.com/bitnami/charts/eb5f9a9513d987b519f0ecd732e7031241c50328/bitnami
"bitnami-pre-2022" has been added to your repositories

It would be fantastic if someone could get this added to the readme.

Answer 5 · 2022-06-02T14:31:37.000Z

Instead of removing all versions older than 6 months, would it be possible to keep the last major/minor versions older than 6 months, eg: mongodb-9.2.6.

Answer 6 · 2022-06-02T15:10:04.000Z

Hi @potiuk, we apologise for the inconvenience. The Helm charts you built & distributed previously still are working properly for users, the previous version of the Helm chart is included into the tarball itself.

The part that is affected is the helm package or helm dependency update actions as part of the packaging process. All the charts are publicly available, all versions. The workaround mentioned into the post allows you to continue building the Helm chart with the exact same version.

Answer 7 · 2022-06-02T15:14:53.000Z

Hi @potiuk, we apologise for the inconvenience. The Helm charts you built & distributed previously still are working properly for users, the previous version of the Helm chart is included into the tarball itself.

The part that is affected is the helm package or helm dependency update actions as part of the packaging process. All the charts are publicly available, all versions. The workaround mentioned into the post allows you to continue building the Helm chart with the exact same version.

Why didn't you consider a proper CDN or basically...anything else? You make it seem like this is no big deal and that there is an existing easy path for people. The reality is that a ton of pipelines, dev flows, and deployments have just broken and people are scrambling to find out why. There is no way to spin that this was a good decision as executed - this was done extremely rashly.

Answer 8 · 2022-06-02T15:25:11.000Z

The Url in this code snippet for your fix returns 404

$ helm repo add bitnami-pre-2022 https://raw.githubusercontent.com/bitnami/charts/eb5f9a9513d987b519f0ecd732e7031241c50328/bitnami
"bitnami-pre-2022" has been added to your repositories

Answer 9 · 2022-06-02T15:26:03.000Z

$ helm repo add bitnami-pre-2022 https://raw.githubusercontent.com/bitnami/charts/eb5f9a9513d987b519f0ecd732e7031241c50328/bitnami
"bitnami-pre-2022" has been added to your repositories

@CrisFavero Normal. Because helm will look for an index.yaml in this base url

Answer 10 · 2022-06-02T15:28:19.000Z

The Url in this code snippet for your fix returns 404

$ helm repo add bitnami-pre-2022 https://raw.githubusercontent.com/bitnami/charts/eb5f9a9513d987b519f0ecd732e7031241c50328/bitnami
"bitnami-pre-2022" has been added to your repositories

redacting incorrect info - the above info is correct

Answer 11 · 2022-06-02T15:30:23.000Z

@CrisFavero

A chart repository is an HTTP server that houses an index.yaml file and optionally some packaged charts. When you're ready to share your charts, the preferred way to do so is by uploading them to a chart repository.

So the current workaround simply uses githubusercontent.com as a free webserver to provide that essential/critical index.yaml, that contains the links to the helm charts versions we all need.

Answer 12 · 2022-06-02T15:36:37.000Z

@adawalli if you don't specify the commit hash then you're using the latest index, which does not include older releases. @CrisFavero this URI worked for me: https://raw.githubusercontent.com/bitnami/charts/eb5f9a9513d987b519f0ecd732e7031241c50328/bitnami

Answer 13 · 2022-06-02T15:38:00.000Z

@adawalli if you don't specify the commit hash then you're using the latest index, which does not include older releases. @CrisFavero this URI worked for me: https://raw.githubusercontent.com/bitnami/charts/eb5f9a9513d987b519f0ecd732e7031241c50328/bitnami

Thanks - you are correct - I redacted my previous comment to prevent confusion

Answer 14 · 2022-06-02T15:38:24.000Z

In order to list old versions you need to use a previous commit, see below the differences:

$ helm repo list
NAME            	URL
bitnami         	https://charts.bitnami.com/bitnami
bitnami-index   	https://raw.githubusercontent.com/bitnami/charts/index/bitnami
bitnami-pre-2022	https://raw.githubusercontent.com/bitnami/charts/pre-2022/bitnami

bitnami (latest 6 months):

$ helm search repo bitnami/common -l
NAME          	CHART VERSION	APP VERSION	DESCRIPTION
bitnami/common	1.15.1       	1.15.1     	A Library Helm Chart for grouping common logic ...
bitnami/common	1.14.2       	1.14.2     	A Library Helm Chart for grouping common logic ...
bitnami/common	1.14.1       	1.13.1     	A Library Helm Chart for grouping common logic ...
bitnami/common	1.14.0       	1.13.1     	A Library Helm Chart for grouping common logic ...
bitnami/common	1.13.1       	1.13.1     	A Library Helm Chart for grouping common logic ...
bitnami/common	1.13.0       	1.13.0     	A Library Helm Chart for grouping common logic ...
bitnami/common	1.12.0       	1.12.0     	A Library Helm Chart for grouping common logic ...
bitnami/common	1.11.3       	1.11.3     	A Library Helm Chart for grouping common logic ...
bitnami/common	1.11.2       	1.11.2     	A Library Helm Chart for grouping common logic ...
bitnami/common	1.11.1       	1.11.1     	A Library Helm Chart for grouping common logic ...
bitnami/common	1.11.0       	1.11.0     	A Library Helm Chart for grouping common logic ...
bitnami/common	1.10.4       	1.10.0     	A Library Helm Chart for grouping common logic ...

bitnami-index, same as bitnami (latest 6 months), but using GitHub to serve it:

$ helm search repo bitnami-index/common -l
NAME                	CHART VERSION	APP VERSION	DESCRIPTION
bitnami-index/common	1.15.1       	1.15.1     	A Library Helm Chart for grouping common logic ...
bitnami-index/common	1.14.2       	1.14.2     	A Library Helm Chart for grouping common logic ...
bitnami-index/common	1.14.1       	1.13.1     	A Library Helm Chart for grouping common logic ...
bitnami-index/common	1.14.0       	1.13.1     	A Library Helm Chart for grouping common logic ...
bitnami-index/common	1.13.1       	1.13.1     	A Library Helm Chart for grouping common logic ...
bitnami-index/common	1.13.0       	1.13.0     	A Library Helm Chart for grouping common logic ...
bitnami-index/common	1.12.0       	1.12.0     	A Library Helm Chart for grouping common logic ...
bitnami-index/common	1.11.3       	1.11.3     	A Library Helm Chart for grouping common logic ...
bitnami-index/common	1.11.2       	1.11.2     	A Library Helm Chart for grouping common logic ...
bitnami-index/common	1.11.1       	1.11.1     	A Library Helm Chart for grouping common logic ...
bitnami-index/common	1.11.0       	1.11.0     	A Library Helm Chart for grouping common logic ...
bitnami-index/common	1.10.4       	1.10.0     	A Library Helm Chart for grouping common logic ...

bitnami-pre-2022, old versions:

helm search repo bitnami-pre-2022/common -l
NAME                   	CHART VERSION	APP VERSION	DESCRIPTION
bitnami-pre-2022/common	1.14.2       	1.14.2     	A Library Helm Chart for grouping common logic ...
bitnami-pre-2022/common	1.14.1       	1.13.1     	A Library Helm Chart for grouping common logic ...
bitnami-pre-2022/common	1.14.0       	1.13.1     	A Library Helm Chart for grouping common logic ...
bitnami-pre-2022/common	1.13.1       	1.13.1     	A Library Helm Chart for grouping common logic ...
bitnami-pre-2022/common	1.13.0       	1.13.0     	A Library Helm Chart for grouping common logic ...
bitnami-pre-2022/common	1.12.0       	1.12.0     	A Library Helm Chart for grouping common logic ...
bitnami-pre-2022/common	1.11.3       	1.11.3     	A Library Helm Chart for grouping common logic ...
bitnami-pre-2022/common	1.11.2       	1.11.2     	A Library Helm Chart for grouping common logic ...
bitnami-pre-2022/common	1.11.1       	1.11.1     	A Library Helm Chart for grouping common logic ...
bitnami-pre-2022/common	1.11.0       	1.11.0     	A Library Helm Chart for grouping common logic ...
bitnami-pre-2022/common	1.10.4       	1.10.0     	A Library Helm Chart for grouping common logic ...
bitnami-pre-2022/common	1.10.3       	1.10.0     	A Library Helm Chart for grouping common logic ...
...
bitnami-pre-2022/common	0.2.4        	0.2.4      	A Library Helm Chart for grouping common logic ...
bitnami-pre-2022/common	0.2.3        	0.2.3      	A Library Helm Chart for grouping common logic ...
bitnami-pre-2022/common	0.2.2        	0.2.2      	A Library Helm Chart for grouping common logic ...
bitnami-pre-2022/common	0.2.1        	0.2.1      	A Library Helm Chart for grouping common logic ...
bitnami-pre-2022/common	0.2.0        	0.1.0      	A Library Helm Chart for grouping common logic ...
bitnami-pre-2022/common	0.1.1        	0.1.0      	A Library Helm Chart for grouping common logic ...
bitnami-pre-2022/common	0.1.0        	0.1.0      	A Library Helm Chart for grouping common logic ...

Answer 15 · 2022-06-02T16:15:54.000Z

@adawalli the information you provided seems to be correct for both cases:

When first trying the workaround to add the pre-2022 repo I got the following results:

$ helm repo add bitnami-pre-2022 \
  https://raw.githubusercontent.com/bitnami/charts/eb5f9a9513d987b519f0ecd732e7031241c50328/bitnami

Error: looks like 
"https://raw.githubusercontent.com/bitnami/charts/eb5f9a9513d987b519f0ecd732e7031241c50328/bitnami" 
is not a valid chart repository or cannot be reached: 
error unmarshaling JSON:
  while decoding JSON: 
    invalid character 'd' looking for beginning of object key string

When retrying the exact same command at a later point in time I got this result:

"bitnami-pre-2022" has been added to your repositories

Right now we can access the required charts again, but since there was different behavior this seems a bit unstable to me 🤔 .

Answer 16 · 2022-06-02T17:07:27.000Z

$ helm repo add bitnami-pre-2022 https://raw.githubusercontent.com/bitnami/charts/eb5f9a9513d987b519f0ecd732e7031241c50328/bitnami
"bitnami-pre-2022" has been added to your repositories

Isn't this just moving the problem from CloudFront to Github? It's a large file no matter who's serving it.

Answer 17 · 2022-06-02T17:12:15.000Z

Oh my god, seriously that you did it?

Answer 18 · 2022-06-02T17:37:09.000Z

Was there truly no way to negotiate with AWS on the bandwidth limits?

At the very least, as @GMZwinge said, you should be providing the latest of each historical major release.

Answer 19 · 2022-06-02T18:30:28.000Z

What's going on? Just today started getting:

Error: can't get a valid version for repositories postgresql. Try changing the version constraint in Chart.yaml

Chart.yaml is:

  - name: postgresql
    version: 10.3.18
    repository: "@bitnami"

This is absurd, can't just break everything for people. Additionally I'm not going to add a helm repo pointing to a GitHub raw commit.

$ helm repo add bitnami https://raw.githubusercontent.com/bitnami/charts/eb5f9a9513d987b519f0ecd732e7031241c50328/bitnami

Bitnami this is horrible developer experience.

Answer 20 · 2022-06-02T18:59:43.000Z

Hi @potiuk, we apologise for the inconvenience. The Helm charts you built & distributed previously still are working properly for users, the previous version of the Helm chart is included into the tarball itself.

The part that is affected is the helm package or helm dependency update actions as part of the packaging process. All the charts are publicly available, all versions. The workaround mentioned into the post allows you to continue building the Helm chart with the exact same version.

Yep. Thanks for the information - this is indeed the case., which I've learned since writing it. So the problem is indeed limited to people who are developing the charts, CI, development pipelines and generally anyone adventorous etc. Which is stil quite a big deal to get overnight change like that. This seriously undermines usefulness of using bitnami charts in the future for any serious project. But maybe that's what you wanted to achieve - this is a free offering which you can't monetize. Travis CI did similar things in the past by deprioritising free OSS support and maybe won it in the "commercial" front but lost big time (mainly to GitHub Actions) on the OSS front. That's really business not technical decision that you probably considered while making the change.

For us the decision is made - we will simply vendor your chart in. I actually even started to discuss it day before yesterday as we saw the flakiness - but now we accelerate this move.

Also one more comment and constructive suggestion. Surprises like that might easily break reputation. Maybe reputation in the free/OSS is not the most important for VMware-owned company, but I think if you want to really solve the problem, and still keep the free offreing "serious" for the free/OSS development users, you should simply deliver each chart separately as separate repository (or basically a sub-folder of the main folder that has separate index).

This is the only sustainable way to make it works long term and much more in-line with Helm 3 approach where they recognized the problem of centralized repos. You should follow their blueprint.

The cncf foundation actually made the decision about getting rid of the centralised repo and even made a deliberate effort to reach out to the community (we got plenty of discussions with cncf folks about it) to prepare everyone for the change well in advance. This change went really went and smooth and I think that was a model approach to change management. You probably know about it but for the passers-by this is just one of the blog posts where they warned about the change.
https://www.cncf.io/blog/2020/10/07/important-reminder-for-all-helm-users-stable-incubator-repos-are-deprecated-and-all-images-are-changing-location/

Answer 21 · 2022-06-02T19:25:57.000Z

Also, the suggested link with all versions is returning 404 as well:

https://raw.githubusercontent.com/bitnami/charts/eb5f9a9513d987b519f0ecd732e7031241c50328/bitnami

Any recommendations? Need to get this resolved, breaking all of our deployments.

Answer 22 · 2022-06-02T19:45:56.000Z

@nodesocket The suggested link is the repo URL you add via a helm repo command or in your declarative infra.

# This will 404 if you visit it in the browser
$ helm repo add https://raw.githubusercontent.com/bitnami/charts/eb5f9a9513d987b519f0ecd732e7031241c50328/bitnami/

Helm will then reach out to https://raw.githubusercontent.com/bitnami/charts/eb5f9a9513d987b519f0ecd732e7031241c50328/bitnami/index.yaml, which returns a 200

Answer 23 · 2022-06-02T19:53:03.000Z

Now that @potiuk has opened my eyes to the monetization topic(thank you very much for that, this is something I always forget), maybe this is some crazy decision related to vmware aquisition by broadcom? This is conspiracy theory, I know, but yeah, money in the CDN mentioned in the discussion maybe?

Answer 24 · 2022-06-02T19:59:55.000Z

@thomasv314 thanks. I tried the following:

$ helm repo remove bitnami
"bitnami" has been removed from your repositories

$ helm repo add bitnami https://raw.githubusercontent.com/bitnami/charts/eb5f9a9513d987b519f0ecd732e7031241c50328/bitnami/
"bitnami" has been added to your repositories

$ helm repo update

But now getting:

Error: no cached repository for helm-manager-54d2620bbb6f1bb3f35d4c7f945bfa25077949488dcbb0a4d01c90f2c35baa59 found. (try 'helm repo update'): open /home/ubuntu/.cache/helm/repository/helm-manager-54d2620bbb6f1bb3f35d4c7f945bfa25077949488dcbb0a4d01c90f2c35baa59-index.yaml: no such file or directory

Running Helm 3.8.2:

$ helm version
version.BuildInfo{Version:"v3.8.2", GitCommit:"6e3701edea09e5d55a8ca2aae03a68917630e91b", GitTreeState:"clean", GoVersion:"go1.17.5"}

Answer 25 · 2022-06-02T21:02:56.000Z

Now that @potiuk has opened my eyes to the monetization topic(thank you very much for that, this is something I always forget), maybe this is some crazy decision related to vmware aquisition by broadcom? This is conspiracy theory, I know, but yeah, money in the CDN mentioned in the discussion maybe?

I wouldn't think of it as "conspiracy" theory. I can totally relate to "free offering having no guarantees" - no complaints on it. And if that's a business decision to be less "free users" friendly, so be it - this perfectly understandable, just painful for those who put their trust in the free service (wiithout any guarantees of course). As an OSS maintainer - I can totally relate to that but while we provide no guarantees, we at least document and agree to policies that we as "free software maintainers under the Apache Software Foundation umbrella" can communicate to our users well in advance (and yeah - we also relentlessly refuse any kind of support when someone "demands" free support beyond those policies).

So this is more the matter of some kind of expectations the users have - commmunicating them and sticking to it. If they are undocumented, there are some implicit assumptions. Changing (or actually introducing) policies overnight without earlier communication and warnings which are effective immediately is one of the worst ways I can imagine to get strong, supportive community and user's trust. That's it.

Reputation can be built for years and lost overnight (especially if there is a lack of self-reflection and responding to your users).

Answer 26 · 2022-06-02T21:19:16.000Z

Now, breaking on RabbitMQ, even with using helm repo add bitnami https://raw.githubusercontent.com/bitnami/charts/eb5f9a9513d987b519f0ecd732e7031241c50328/bitnami/.

The Chart.yaml for our RabbitMQ is:

annotations:
  category: Infrastructure
apiVersion: v2
appVersion: 3.9.8
dependencies:
  - name: common
    repository: https://charts.bitnami.com/bitnami
    tags:
      - bitnami-common
    version: 1.x.x
  - name: rabbitmq
    version: 8.24.1
    repository: https://charts.bitnami.com/bitnami
description: Open source message broker software that implements the Advanced Message Queuing Protocol (AMQP)
engine: gotpl
home: https://github.com/bitnami/charts/tree/master/bitnami/rabbitmq
icon: https://bitnami.com/assets/stacks/rabbitmq/img/rabbitmq-stack-220x234.png
keywords:
  - rabbitmq
  - message queue
  - AMQP
maintainers:
  - email: containers@bitnami.com
    name: Bitnami
name: rabbitmq
sources:
  - https://github.com/bitnami/bitnami-docker-rabbitmq
  - https://www.rabbitmq.com
version: 8.24.1

Error message is:

Error: can't get a valid version for repositories rabbitmq. Try changing the version constraint in Chart.yaml

$ helm repo list
NAME      	URL
stable    	https://charts.helm.sh/stable
minio     	https://helm.min.io
datadog   	https://helm.datadoghq.com
fusionauth	https://fusionauth.github.io/charts
bitnami   	https://raw.githubusercontent.com/bitnami/charts/eb5f9a9513d987b519f0ecd732e7031241c50328/bitnami/

Answer 27 · 2022-06-02T22:57:01.000Z

@carrodher any idea why RabbitMQ is still broken even when using bitnami https://raw.githubusercontent.com/bitnami/charts/eb5f9a9513d987b519f0ecd732e7031241c50328/bitnami/?

- name: rabbitmq
    version: 8.24.1
    repository: https://charts.bitnami.com/bitnami

Answer 28 · 2022-06-02T23:51:20.000Z

Breaking an index.yaml seems not great: it's kind of like breaking an API version without a major version change. Some alternatives

Give each subchart their own index.yaml (charts.bitnami.com/postgres/index.yaml)
Keep the old index.yaml, deprecate that location (no longer give it updates), and put newer charts on a different URL with different guarantees about longevity
Reduce the size of index.yaml by removing everything not necessary (icon, annotations, created, etc)

Answer 29 · 2022-06-03T00:57:08.000Z

Keep the old index.yaml, deprecate that location (no longer give it updates), and put newer charts on a different URL with different guarantees about longevity

Yes 💯 .

Answer 30 · 2022-06-03T03:11:13.000Z

This is absolutely cooked

Answer 31 · 2022-06-03T04:52:06.000Z

Just out of curiosity, why not just use compression...

wget https://raw.githubusercontent.com/bitnami/charts/eb5f9a9513d987b519f0ecd732e7031241c50328/bitnami/index.yaml

If you take that file and compress to

.gz with regular gunzip you'll get a 1.1MB file
.xz with xz -9 gets you down to 852k

So basically, this doesn't seem to be a problem if compression is supported. Seems you could host it on another mirror and push out a minor version of Helm that supports compression and you'd be good. 852k is basically the size of a single-page app wth Angular or React.

I also haven't looked, but if the client supports gzip you could just rely on transparent http gziping with something like mod_deflate.

Answer 32 · 2022-06-03T05:20:21.000Z

Hi. Does Helm supports being redirected during index.yaml fetch? If so, would it be possible to redirect the request for index.yaml from the old location to the pre-2022 commit endpoint? That way all old versions would still be available without breaking things.

Answer 33 · 2022-06-03T09:54:20.000Z

@carrodher Can you please share a clear plan on where you guys are going with this issue.
We ned to plan accordingly in order to a) get this back up and running, b) make sure we choose to right measure for the future.
Also sharing a timeline for "back-to-normal" would greatly help

Answer 34 · 2022-06-03T10:32:50.000Z

Why not use any of the hosting provides that offers free hosting for open source projects? Perhaps host the helm index on github pages?

Answer 35 · 2022-06-03T11:18:27.000Z

How could you possible introduce such a breaking change? This is absolutely pathetic.

Answer 36 · 2022-06-03T12:23:34.000Z

You are breaking too many builds. postgresql 10.9.5 just disappeared and I had to re-write the entire values file with the new one forcing us to take care of breaking changes. It is dead simple to maintain an index for at least LTS releases for the underlying software versions.

Answer 37 · 2022-06-03T13:14:36.000Z

@carrodher the right thing to do here is to immediately roll back this change, and work out how to proceed afterwards - occasional timeouts are a better situation than total breakage. Leaving the repo in the current state with users' production pipelines broken is untenable - even if there's a wonky workaround (relying on Github to act as your CDN for a pinned commit, instead of fixing the prod CDN), that's likely thousands of builds that need to be updated to continue functioning, once those affected can actually work out why their builds exploded.

Answer 38 · 2022-06-03T13:22:39.000Z

Rollback please, is this a joke??

Answer 39 · 2022-06-03T13:23:28.000Z

Maybe they're freeing up enough space for those ARM charts we've been waiting for.

Answer 40 · 2022-06-03T13:34:49.000Z

Constructive suggestion: perhaps provide OCI Helm repo? That would completely bypass index.yaml issue. Or maintain both - old-style repo trimmed down to 6 month or however long that should be to keep thing reasonable and OCI repo with everything

Alternatively perhaps going the route suggested earlier: that for anything older than 6mo keep the most recent minor release for each major one?

Answer 41 · 2022-06-03T14:04:19.000Z

We're still waiting for you to roll this back...

Answer 42 · 2022-06-03T14:29:56.000Z

Just out of curiosity, why not just use compression...
wget https://raw.githubusercontent.com/bitnami/charts/eb5f9a9513d987b519f0ecd732e7031241c50328/bitnami/index.yaml
If you take that file and compress to

.gz with regular gunzip you'll get a 1.1MB file

.xz with xz -9 gets you down to 852k

So basically, this doesn't seem to be a problem if compression is supported. Seems you could host it on another mirror and push out a minor version of Helm that supports compression and you'd be good. 852k is basically the size of a single-page app wth Angular or React.

I also haven't looked, but if the client supports gzip you could just rely on transparent http gziping with something like mod_deflate.

Indeed, compression can easily be enabled in CloudFront

Even using the GitHub hosted commit as the Helm repo, I am still unable to helm dependency update RabbitMQ version 8.24.1 for some strange reason. Please roll this back!!!

Answer 43 · 2022-06-03T16:10:34.000Z

So basically, this doesn't seem to be a problem if compression is supported

Thanks for the suggestion but, even compression is supported by CloudFront, it is not supported by Helm client right now.

Answer 44 · 2022-06-03T17:37:13.000Z

To anybody else, the reason that postgresql worked but rabbitmq did not was in the Chart.yaml for postgresql is was using:

  - name: postgresql
    version: 10.3.18
    repository: "@bitnami"

Note, the reference to the repository as @bitnami. However rabbitmq chart was hardcoding the repository url as:

  - name: rabbitmq
    version: 8.24.1
    repository: https://charts.bitnami.com/bitnami

To fix the problem, you must also update Chart.yaml for each chart and use the dynamic repository reference of @bitnami. Thus the new rabbitmq Chart.yaml should look like:

  - name: rabbitmq
    version: 8.24.1
    repository: "@bitnami"

This still does not address the core problem of having to do:

$ helm repo remove bitnami
"bitnami" has been removed from your repositories

$ helm repo add bitnami https://raw.githubusercontent.com/bitnami/charts/eb5f9a9513d987b519f0ecd732e7031241c50328/bitnami/
"bitnami" has been added to your repositories

$ helm repo update

Answer 45 · 2022-06-03T17:57:32.000Z

What about using e.g. CloudFlare? I‘m sure they can accomodate all your needs…

Answer 46 · 2022-06-03T17:58:45.000Z

What is the suggestion going forward? If we apply the workaround won't we be in the same situation again in 6 months (some of our in-use charts disappear from the current index.yaml)? I'm looking for some way to be able to update my helm charts versions in the future without changing repositories. I could set up a local service of some kind if that would work.

Answer 47 · 2022-06-03T18:02:15.000Z

Can anybody document the process of setting up our own bitnami chart mirror? Is it a simple as:

serve https://raw.githubusercontent.com/bitnami/charts/eb5f9a9513d987b519f0ecd732e7031241c50328/bitnami/index.yaml
update helm repo for bitnami:

$ helm repo remove bitnami

$ helm repo add bitnami https://our-bitnami-chart-mirror.io

$ helm repo update

Or do we also need all the contents of all charts not just the index.yaml?

Answer 48 · 2022-06-03T18:46:29.000Z

@nodesocket I think that will only serve the fixed-point-in-time index. If you want to serve all the things locally, it would be something more like:

git pull upstream master
helm repo index
helm repo serve

That should get your local repo running. Then switching over to use that is as you described in your step 2. I haven't actually set this up yet, but I'm a-fixin' to this weekend if this isn't all reverted back to a functioning state.

That should all work fine until someone decides to delete git history because it takes a long time to clone...

Answer 49 · 2022-06-03T19:41:31.000Z

@jfklingler thanks so much for the info. Can you report back after your work this weekend? I have a feeling I'm going to need to host my own mirror for Bitnami Charts as we need legacy versions and using a GitHub commit as a CDN is not a solution.

Answer 50 · 2022-06-03T20:13:52.000Z

It appears the latest version 17.2.3 of the bitnami/kafka chart isn't in the published index. The older 17.1.0 version is the most recent one I can find in the index.

Is this intentional or am I misunderstanding the steps that were taken as outlined above? Why wouldn’t the latest chart version be in the index, which is unfortunately only supposed to contain versions as recent as 6 months back?

Answer 51 · 2022-06-03T22:19:16.000Z

What is the suggestion going forward? If we apply the workaround won't we be in the same situation again in 6 months (some of our in-use charts disappear from the current index.yaml)? I'm looking for some way to be able to update my helm charts versions in the future without changing repositories. I could set up a local service of some kind if that would work.

This is why this change cannot stand - basically what's been implemented is a rolling window of chart breakage. Every time a chart exceeds the 6 month window, someone's build will break if it relies on the index, and the only way to solve it is for users to create a new repo entry for every single bitnami chart that they consume, pointing to the specific commit for the chart index that includes the required version - this is clearly completely untenable, and makes the index almost useless, with upgrades being a silly dance of finding a commit in the history that has the version you need to upgrade to.

Please stop the damage now by reverting, so we can work out how to move forward.

Answer 52 · 2022-06-04T02:29:43.000Z

There are a lot of smart ops and devops people on this thread I assume (maybe some even work for CDN companies) that can help with the issue. I can only speak for myself, but I represent and own a company and we are willing to pay money to have Bitnami support and treat containers and Helm charts as enterprise assets. Willing to pay monthly/annual fees as long as the uptime, support, (arm64 support, clearing throat) gets executed on.

There are entire companies that provide way less value than Bitnami yet literally print money. It's time we as developers stop being so cheap and support projects like Bitnami assuming they are willing to devote the engineering and capital resources required.

Point being, I'm open to breaking the free and open source model as long as the quality of Bitnami is raised and uptime and confidence is addressed.

Answer 53 · 2022-06-04T04:28:59.000Z

I appreciate the challenge, however with all due respect, a one size fits all 6 months window is not a sensible approach.

Please consider

Reverting this change
Maintain LTS versions, e.g. always have last 3 minor versions of chart indexed & available
Define a sensible policy for housekeeping your repo index (eg less frequent releases)

Unfortunately would feel compelled to consider finding a more stable alternative if this is not fixed.

Thanks!

As noted by @potiuk

This seriously undermines usefulness of using bitnami charts in the future for any serious project.

@carrodher please realize this change is undermining trust in bitnami, and it will be hard o build it back if this issue is not addressed in a senisble manner. Bitnami is used because developer find it provided good value, they are the ones to make the argument to buy a license. In absence of value; many will juat move on.

Answer 54 · 2022-06-04T04:55:48.000Z

A potential long-term solution would be to generate repositories for each chart, that serve just that chart and its revisions - this would drastically reduce the required bandwidth, since users would only fetch the data for the specific charts they're interested in.

Obviously this would also be a breaking change, and require some engineering effort to implement, however this could be done in parallel with serving the existing legacy full-index in the mean time (stats would provide data on when it might be sensible to deprecate the legacy index).

Alternatively, just finding a CDN that is willing to carry the traffic would be simpler for all involved in the nearer term.

Discussions around ways to manage the problem really need to happen before breaking the world though.

Answer 55 · 2022-06-04T05:38:21.000Z

CloudFlare literally caches files up to 512 MB for free:

Cloudflare cacheable file limits:
Free, Pro and Business customers have a limit of 512 MB.

https://developers.cloudflare.com/cache/about/default-cache-behavior/

Answer 56 · 2022-06-04T07:08:45.000Z

@mprimeaux wrote:
It appears the latest version 17.2.3 of the bitnami/kafka chart isn't in the published index. The older 17.1.0 version is the most recent one I can find in the index.

Is this intentional or am I misunderstanding the steps that were taken as outlined above? Why wouldn’t the latest chart version be in the index, which is unfortunately only supposed to contain versions as recent as 6 months back?

This is a totally different topic. As pointed out in the CONTRIBUTING guidelines:

NOTE: Please note that, in terms of time, may be a slight difference between the appearance of the code in GitHub and the chart in the registry.

This is caused because we test all the charts on top of different k8s clusters (TKG, IKS, AKS, GKE, ...) before publishing them into the registry.

The case you mention is already published, see

$ helm repo update bitnami
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "bitnami" chart repository
Update Complete. ⎈Happy Helming!⎈

$ helm search repo bitnami/kafka
NAME         	CHART VERSION	APP VERSION	DESCRIPTION
bitnami/kafka	17.2.3       	3.2.0      	Apache Kafka is a distributed streaming platfor...

Also please note the change in the index.yaml was, at least at this moment, one-time action. Entries older than a specific date (6 months ago) were cleaned in a single PR, there is not a cron job doing it.

Answer 57 · 2022-06-04T09:39:46.000Z

Also please note the change in the index.yaml was, at least at this moment, one-time action. Entries older than a specific date (6 months ago) were cleaned in a single PR, there is not a cron job doing it.

I think you totally miss the expectations here @carrodher of people who used the free chart provided by bitnami. It would have been way better if people knew what to expect, because they would have at least a chance to prepare their workflows for that..

This above statement of yours is the worst nighmare of anyone who would use the charts. It basically means that anyone using the repo should expect that similar action might happen any time, without any notice or warning, brreaking their workflows.

Is this the right way I am reading that statement ?

Is this what the "bitnami chart" free offering policy is now?

If so, it makes it useless. But maybe that is the final goal you want to achive. It definitely looks like your actions goal is to make sure the "non-paying" users stop using bitnami.

And that's fine. Nobody expects you to do that for free for ever. That was a voluntary choice of bitnami to do so before and theiy build their strength, value (up to the moment it was acquired) - so the "free model" was definitely working for you before to build the company and make an exit.

And it's ok if you decide to stop (ideally with some warning/notice) providing good service for non-paying (often open-source) users.

I think however it would be much more transparent and sincere if you just openly state it. It would be much more honest towards all those "non-paying" users who helped you to build the value and sell the company.

Answer 58 · 2022-06-04T10:30:58.000Z

@potiuk this is a totally unrelated conversation. @mprimeaux asked about a recent version not appearing in the index.yaml. I clarified that this is not related to the truncation of the index.yaml but related to how our automated test & release pipeline works.

What I said is that there is not any automation truncating this file that is executed every X time. It was a one-time action executed some days ago, so it is not possible that a recent Helm chart version not appearing in the index.yaml was caused because of the truncation of the index.yaml since there is nothing truncating this file again.

When this file needs to be truncated again in the future or if there is any automation implemented to truncate it automatically every X time, it would be communicated in advance, but this is not the case right now. Any other assumption without knowing the whole context of the previous answer is just that, assumptions.

Answer 59 · 2022-06-04T11:03:00.000Z

Also please note the change in the index.yaml was, at least at this moment, one-time action. Entries older than a specific date (6 months ago) were cleaned in a single PR, there is not a cron job doing it.

This is critical information, and is not what is implied by the title (retention policy), and content (we will reduce the size of the index.yaml by removing some old versions and keeping all versions for a period of time (6 months)) of this issue, both of which imply that this would be an ongoing approach.

That this is not the case is positive on the one hand, since the ongoing impact should be less drastically bad, though at least at this moment, one-time action does imply that this may be automated at some stage in the future. On the other hand, this change is impacting a not-insignificant number of people today.

Uncertainty about how this may be handled going forward, the drastic action taken by implementing this now without adequate communication, and the lack of any official response to the concerns raised across all the related issues for this change are cause for significant concern for anyone who has in the past relied on the bitnami chart repo for production workloads - it is very difficult to justify placing trust in this repository going forward.

This is quite unfortunate as it undermines all the great work that has gone into making this an otherwise wonderful resource.

Answer 60 · 2022-06-04T12:36:31.000Z

@carrodher Thanks for your clarification. At the time, I wasn't sure if the issue we were experiencing was related to the truncation.

Answer 61 · 2022-06-04T15:03:41.000Z

When this file needs to be truncated again in the future or if there is any automation implemented to truncate it automatically every X time, it would be communicated in advance, but this is not the case right now. Any other assumption without knowing the whole context of the previous answer is just that, assumptions.

@carrodher - the assumption are caused by bitnami actions and complete lack of explanation, setting clear policy and even attempting to acknowledge that you understand problems of your users. One can see that as super arrogant approach. One that shows that bitnami cannot be trusted that they see their non paying users as such even though they build their value banking on those many users and popularity of bitnami built this way.

I personally sse that as rather unprofessional approach and lack of self-reflection and recognising that your actions are relly causing great problems to the users who trusted bitnami. After those few days when Bitnami was not even able to admit "yes we've done that hastily, we are working on solving the problem and responding to our user concernrs" we just see complete lack of communication, empathy, lack of self-reflection.

This is very, very bad for the brand, even if Bitnami wanted to make it useless for non-paying users, they probably chose the worst possible approach. Or if they did that unknowingly and without realising the consequence that means that this was an extremely unprofessional behaviour (which makes even more damage to Bitnami brand)

Pick your poison. But I am afraid - both are basically sending Bitnami's brand's reputation to the bottom of the ocean - after Bitnami even failed to react to the reactions of their users.

Answer 62 · 2022-06-05T00:45:23.000Z

@carrodher said

When this file needs to be truncated again in the future or if there is any automation implemented to truncate it automatically every X time, it would be communicated in advance, but this is not the case right now.

Well, that's scary. Please acknowledge you understand that truncating the index file in absence of a policy that preserves latest stable minor versions is just a bad engineering decision in all dimensions. Also, and perhaps even more worrying, it is totally at odds with the bitnami stated philosophy. Indeed this change is breaking the bitnami brand's very promise, as stated on your webpage (emphasis by me).

Bitnami makes it easy to get your favorite open source software up and running on any platform, including your laptop, Kubernetes and all the major clouds. In addition to popular community offerings, Bitnami, now part of VMware, provides IT organizations with an enterprise offering that is secure, compliant, continuously maintained and customizable to your organizational policies.

Answer 63 · 2022-06-06T09:34:38.000Z

I humbly suggest splitting each chart into it's own repository, which will greatly reduce the file size, and make more sense from a lot of points of view :)

Answer 64 · 2022-06-06T13:45:05.000Z

@nodesocket So I gave my local bitnami repo the old college try and got bogged down in a jq/yq mess. Given the additional clarification that they've given, namely that 6 months is not a rolling window but an arbitrary cutoff, I ultimately decided it would be easier to add a "bitnami-old" repo in addition to the new, slimmer "bitnami" repo and just update my helm releases accordingly. I'm using these chart through flux so updating the resources it was complaining about was pretty trivial for my case, but I'm sure it isn't for everyone. Sorry I don't have a better solution for you.

Answer 65 · 2022-06-06T15:46:08.000Z

This is an extremely disappointing move that will add a lot of unnecessary engineering effort on many teams out there. It also make us question the trust we put into using bitnami provided charts. Will definitely think twice before choosing to use bitnami chart next time.

Edit: +1 for transparency on the change here; -1 for the new retention policy

Answer 66 · 2022-06-06T16:12:58.000Z

This will also break GitLab's Auto DevOps feature. In case anybody runs into the same issue (took me a few hours to track down and eventually "fix"/work around). If you see this error in your deployment job:

Error: failed to download "bitnami/postgresql" at version "8.2.1"

Then you need to update to the latest auto-deploy-image version of GitLab by setting the CI/CD variable AUTO_DEPLOY_IMAGE_VERSION to v2.28.2 in your project. The workaround pins the helm repo to use the old bitnami index.

Answer 67 · 2022-06-06T16:24:03.000Z

Even if you have to trim it down any thoughts to at least including the latest major.minor versions for each set in the default index.yaml . Some of these artifacts that are being removed are less than a year old.

Answer 68 · 2022-06-06T16:29:39.000Z

im afraid to post this, but i dont think the recent pruning of chart versions in the main branch manifest solved the original issue of the CDN timing out or rejecting

    Exception: invoke of kubernetes:helm:template failed: invocation of kubernetes:helm:template returned an error: failed to generate YAML for specified Helm chart: failed to pull chart: looks like "https://charts.bitnami.com/bitnami" is not a valid chart repository or cannot be reached: stream error: stream ID 1; INTERNAL_ERROR; received from peer

Answer 69 · 2022-06-06T16:44:02.000Z

Even if you have to trim it down any thoughts to at least including the latest major.minor versions for each set in the default index.yaml . Some of these artifacts that are being removed are less than a year old.

@datadidit Please, read the article. Nothing has been removed.

Answer 70 · 2022-06-06T16:48:20.000Z

Even if you have to trim it down any thoughts to at least including the latest major.minor versions for each set in the default index.yaml . Some of these artifacts that are being removed are less than a year old.

@datadidit Please, read the article. Nothing has been removed.

Correct, I mistyped above thanks for the correction. But still curious about possibly keeping the latest <major.minor> version in the default index.yaml . Which would allow teams not to need to make a major version change of the dep and or need to add the additional repo to get the old version of the chart.

Answer 71 · 2022-06-06T16:49:58.000Z

@datadidit We are working to accommodate all the feedback received. We will get back here soon.

Answer 72 · 2022-06-06T18:35:18.000Z

Hi everyone,

Me and the rest of the Bitnami team sincerely apologize for the issues you experimented related to the decision of non-indexing an old version of Helm charts into the main index.yaml file.

The main issue that triggered that action is that index.yaml is getting thousands of terabytes of download traffic per month, the AWS CDN itself was increasing exponentially the number of errors every week and, that rate was so high that was also affecting our own release pipelines.

This decision was risky and not communicated properly with enough anticipation, sorry about it. We underestimated the impact of the de-indexing older versions even all the versions continue being publicly available.

In order to move on, we will continue with a policy of not indexing versions older than 6 months in the official Helm chart repository https://charts.bitnami.com/bitnami. This index will continue available into a CDN to speed up the transfer and we need a de-listing policy to keep it smaller and to avoid error rates.

We will work on implementing and maintaining a new full Helm chart index file that Bitnami will keep up-to-date with all the releases. This index will be available in GitHub itself, not into the CDN, so users who wants to keep using older versions and the size or speed is not critical can continue using a full index.yaml file or users can easily mirror it. All the Helm charts tgz will continue being publicly available as always.

On the other hand, we are working on fully supporting Helm Charts in OCI registries. That is fully supported by Helm since 3.8.0 version. It is still early to know what would be the next steps but we will share that info whenever we get clarity about it.

Answer 73 · 2022-06-06T19:08:53.000Z

Thanks for at least acknowledging the issue and establishing the policy. At least we know what to expect. Looking forward for the index tha will be maintained and have some solid maintenance policies too.

Answer 74 · 2022-06-06T19:20:14.000Z

@beltran-rubo thanks for the update and clarification. Can you provide documentation/guide on how we can self-host? We may just want to roll our own mirror of Bitnami charts using our own CDN provider.

This index will be available in GitHub itself, not into the CDN, so users who wants to keep using older versions and the size or speed is not critical can continue using a full index.yaml

We are forced to use the old version of index.yaml as we have very specific version constraints for RabbitMQ, Redis, and PostgreSQL that cannot be upgraded (at least easily). However speed and more importantly consistency and reliability are important as these are production clusters. Thus, why we are looking to self host a mirror of Bitnami charts.

Answer 75 · 2022-06-07T00:01:50.000Z

@beltran-rubo thanks for the response, I would have liked to see more information on the the status and progress of the Helm client supporting compression because that seems like a very obvious route to move forward.

Roll out a new helm client that:
1. supports compression (like v3.9.1 this has got to very easy)
2. points to a new chart location https://charts.bitnami.com/v2/bitnami
Call this a "bug" in the ecosystem, and v3.9.1 becomes a mere bug-fix release.

Then, at the very least, you can respond to your clients that you're trying to follow the semver versioning convention/contract with the bug caused by the demands on the CDN being available as a patch release.

You can see my numbers above on the compression gain, and no matter what this is a solid improvement without any breakage on the truncation (smaller size, more data). As others have pointed out too, CloudFlaire has transparent http compression too. This could be enabled seemless without duplicating the end point if the client right now, which doesn't compression, provides appropriate HTTP Accept headers.

As a last note, if all that falls on deaf ears or isn't possible I think at least changing the pruning policy such that the last stable-release of a chart isn't pruned, regardless of age is something that should be addressed. That people are complaining this happened seems like a bug even if Bitnami does prefer the solution they've created.

Answer 76 · 2022-06-07T00:08:05.000Z

@beltran-rubo thanks for the response, I would have liked to see more information on the the status and progress of the Helm client supporting compression because that seems like a very obvious route to move forward.

If you read the upstream Helm issue, the problem is that various hosts send gzipped archives with mime types that cause them to be decoded on the wire, resulting in them failing to be unpacked locally, since they've been pre-decompressed.

Regardless of how this issue has been mismanaged here, bitnami has no control over upstream Helm, and in any case, this does not seem like an easy problem to solve, as it's an interaction between the Go stdlib and specific hosting providers, neither of which are likely to change for this specific problem.

Answer 77 · 2022-06-07T00:25:01.000Z

The main issue that triggered that action is that index.yaml is getting thousands of terabytes of download traffic per month, the AWS CDN itself was increasing exponentially the number of errors every week and, that rate was so high that was also affecting our own release pipelines.

Have you at all considered an alternative CDN, as has been suggested a few times now?

In order to move on, we will continue with a policy of not indexing versions older than 6 months in the official Helm chart repository https://charts.bitnami.com/bitnami. This index will continue available into a CDN to speed up the transfer and we need a de-listing policy to keep it smaller and to avoid error rates.

A rolling 6-month chart index just means that things will be constantly broken for users, for reasons already explained, and that should be obvious.

We will work on implementing and maintaining a new full Helm chart index file that Bitnami will keep up-to-date with all the releases. This index will be available in GitHub itself, not into the CDN, so users who wants to keep using older versions and the size or speed is not critical can continue using a full index.yaml file or users can easily mirror it. All the Helm charts tgz will continue being publicly available as always.

Then everyone who needs a stable chart index will be forced to use this version, and you've just moved the CDN problem to Github.

These are band-aids, not solutions.

Answer 78 · 2022-06-07T01:19:32.000Z

@pdf I reviewed that issue more carefully, and I did miss some that the first time I went through it. You may be interested in response to it here: helm/helm#2916 (comment) I don't think this was an issue with core Go stdlibs. I think the disabling of on the wire compression by helm is probably a mistake and a result of specifically Atlassian's Bitbucket implementation. But I didn't realize that Bitnami wasn't the driving force behind helm, so your point remains. Helm should undo the patch, not work around the broken Atlassian implementation (it's not their problem), and then transparent compression by CloudFlare should just work. But, I understand now, in light of that bad patch (my opinion) to work around a BitBucket bug, and without control of Helm, there is nothing really Bitnami can do regarding compression.

Answer 79 · 2022-06-07T08:28:26.000Z

Hi, thank you for keeping this thread alive. Obviously we have been hit by this change, too. A bit surprising that such a small change can cause such disruption. For production we always package the Helm Chart and its dependencies, so for us is causing issue only on our CI/CD internal pipelines.

With the comments from @beltran-rubo, I suspect what will be noticed is that no-one (hypothetically) will be using the official Helm Chart repo url (https://charts.bitnami.com/bitnami) but instead use the full version hosted on GitHub. Not sure why there is a need for a CDN repo that only contains releases from the past six months. Unless you update your Helm Chart dependencies every 4 months, using the CDN repo could be very erroneous and disruptive.

Answer 80 · 2022-06-07T08:43:28.000Z

Unless you update your Helm Chart dependencies every 4 months, using the CDN repo could be very erroneous and disruptive.

Yep. As I explained before the current "main index" policy makes it essentially useless for any serious use.

I really look forward that btinami will keep their promise and introduce (as @beltran-rubo mentioned in #10539 (comment) ) a usable solution:

We will work on implementing and maintaining a new full Helm chart index file that Bitnami will keep up-to-date with all the releases.

I really hope we all collectively gave Bitnami/VMWare/Broadcom a lot to think about. They broke a lot of the use of their free chart by rush and not-well-thought decisions, without realising consequences. They at least finally acknowledged that. It took 70+ comments and more than 4 days, but they finally did acknowledge they screwed up.

They could have approached it differently, but hopefully they learned and approach it differently in the future and make an offering that will keep their non-paying users happy (or not if they make such a business decision - we cannot force them to).

I think we should give them some breathing space, and simply observe their efforts of trying to build their reputation back. It will be long and process before the community will again trust them, but well, let's give them a chance.

Answer 81 · 2022-06-07T08:45:49.000Z

I am for one closely looking at what is happening here to see how they will manage the situation - the "popcorn" is ready - so far it's been almost like a thriller movie to watch :D

Answer 82 · 2022-06-07T09:15:33.000Z

we are working on fully supporting Helm Charts in OCI registries

@beltran-rubo is there an issue we can track? that would be amazing.

We've been using OCI chart images since the beta for our application charts; they are great.

Answer 83 · 2022-06-07T15:39:08.000Z

~~FYI but your suggested workaround doesn't seems to work with a lot of the tools of the Helm ecosystem (example: helm-releaser & its official GitHub action helm/chart-releaser-action).~~ nvm, it does

Answer 84 · 2022-06-07T16:35:18.000Z

There is a patch pending helm now by @prizov that adds on-the-wire compression to Helm for the package list.

See this post for more information helm/helm#2916 (comment) The results are predictable, from the patch notes.

And the result shows a significant(~14x) reduction for transferred traffic: 1,41mb vs 19,72mb.

Because this enables on-the-wire compression for all package-list end points, the question here is whether or not it would be ok to enable on-the-wire compression in CloudFlare, which should only perform on-the-wire compression for clients that request it. When enabled, this would provide the immediate benefit for newer patched Helms, but older Helm clients would still be downloading the uncompressed version.

I have no idea how often people upgrade their Helm.

Answer 85 · 2022-06-07T21:58:33.000Z

@EvanCarroll nice to see the gzip compression PR, but it would require updating the helm binary on all hosts correct? Not really a solution for us, as our Helm version has to stay pinned at 3.8.2 because 3.9.0 breaks for us on EKS. See helm/helm#11007.

Answer 86 · 2022-06-07T22:26:35.000Z

@nodesocket benefits would rely on each user updating their helm version, so until a critical mass of users updated their helm install, the existing CDN would continue to fail regularly.

Your problem with EKS is solved by updating the client auth version in your kubeconfig, and has nothing to do with Helm specifically (kubectl 1.24 removed support for the deprecated v1alpha1 version): helm/helm#10975 (comment)

Answer 87 · 2022-06-07T22:44:09.000Z

@pdf we don't use awscli but we use aws-iam-authenticator. Our install script currently simply does:

curl -o aws-iam-authenticator https://amazon-eks.s3.us-west-2.amazonaws.com/1.21.2/2021-07-05/bin/linux/amd64/aws-iam-authenticator
sudo chown root:root aws-iam-authenticator
sudo chmod 755 aws-iam-authenticator
sudo mv aws-iam-authenticator /usr/local/bin

Answer 88 · 2022-06-07T23:08:59.000Z

@nodesocket this is not the place to troubleshoot that issue further - there is a solution, you'll need to find it elsewhere though.

Answer 89 · 2022-06-08T02:01:57.000Z

@nodesocket this is not the place to troubleshoot that issue further - there is a solution, you'll need to find it elsewhere though.

I appreciate that we can't solve every individual problem on this issue thread, but just kicking the can saying there's a solution somewhere without any pointers is not helpful. If you don't know if there's a solution, say so. There's no shame in not knowing everything, for sure. If you do know of a solution, provide a link so others in the same situation can find a path.

Answer 90 · 2022-06-08T02:06:53.000Z

I already linked to the generalised solution in my previous comment. Making that work with the helper or whatever may require additional research, this is not the place for that discussion.

Answer 91 · 2022-06-08T02:16:23.000Z

I already linked to the generalised solution in my previous comment. Making that work with the helper or whatever may require additional research, this is not the place for that discussion.

Fair enough. I didn't read enough of the thread history on this particular bit and jumped at the apparent lack of empathy/understanding given how this whole thing started. My fault, I apologize.

Answer 92 · 2022-06-08T02:58:50.000Z

Not really a solution for us, as our Helm version has to stay pinned at 3.8.2 because 3.9.0 breaks for us on EKS. See helm/helm#11007.

Just to be clear, if Helm devs think this is a bug fix for 3.8.2 there is no reason why it has to be stuck in 3.9. There is no reason why it can't go into 3.8 and produce 3.8.3 too. It seems like a bug fix to me. It's not new functionality. We know what patch broke it. We have a bug fix/patch that will fix it. It seems a 3.8.3 with compression on the package list would solve your problem. This is a pretty normal ask too.

The real question here is _if_I Helm upstream obliged and produced a 3.8.3 would Bitnami/CloudFlare find it acceptable, having an upgrade path for people on 3.8.2, but also not being able to force them into upgrading.

I'm not getting into the adoption of 3.9.0 either here, intentionally: I agree that conversation is probably better suited elsewhere.

Answer 93 · 2022-06-08T04:09:53.000Z

Me and the rest of the Bitnami team sincerely apologize for the issues

Thanks for recognizing this was a bad move. Good to know bitnami is listening.

However please note that the issue persists.

a) The work around is not really practical and just shifts the problem from your CDN to github, plus(!) keeps many workflows broken. Not helping and bound to cause more frustration.

b) your "comms fix" is not really a fix either, in part because the comms is not the key to what is amiss here, but more importantly because it just reiterates the same bad approach. We are not critizing the words, but what they mean.

c) please acknowledge the real issue: what may look like a smart move on your end (truncating the index) is a major issue for most of your users bc it totally breaks basic and long-standing assumptions (ie. the index is immutable, as it should be) and replaces them with a broken-by-design policy that is bound to just continue the misery forever. Not a good premise.

In order to move on, we will continue with a policy of not indexing versions older than 6 months in the official Helm chart repository

As stated by @potiuk and many others that is not a solution. A solution should solve the problem, not sugar coat the wrong approach.

Here's my 5 cents. I may be wrong, but hear me out.

IMHO the better solution is to remove intermediate chart versions, say for all but the last three minor versions of each chart. So the index would continue to have:

all entries for the last 3 minor versions of every chart
Just one entry, namely the latest, for each previous chart version

I did not run the numbers, but by my guesstimate this should reduce the index file massively. It still leaves users with the - manageable - caveat of having to update to the latest minor version of their resp. charts.

Any forced upgrade is a hassle but at least would allow users to upgrade their charts in a controlled manner rather than having to scramble to some time consuming index replacement (which in many cases will not solve the issue, eg. due to chart interdependencies referring the official index), or to upgrade all charts to the latest version and keep doing this every so often (which may not even be possible due to pinned versions of the underlying software; in any case upgrading charts can be disrupting in unexpected ways. Think dragons)

The risk to bitnami is that once people realize they have to engage in some major engineering activity to get out of this, with no sensible approach offered by bitnami,, people will start looking elsewhere and may not come back.

Answer 94 · 2022-06-08T05:47:40.000Z

I wish we could stop talking about better ways to truncate an index file. No one does this. Perl does pretty much the same thing with CPAN. You can download their index file here,

http://www.cpan.org/modules/02packages.details.txt.gz

It's 2.3MB compressed. With 249,747 packages, never truncated. Even if your need more meta information, it shouldn't be that much different in an index.

Answer 95 · 2022-06-08T09:34:56.000Z

just for the record: other providers are dealing with this by having a separate "helm repository" per application, and not every application mangled together in one registry. so the index can be fairly smaller, and not everybody is syncing everything they even not need.

for example: https://kubernetes.github.io/ingress-nginx/index.yaml

vs: https://charts.bitnami.com/bitnami/index.yaml

Answer 96 · 2022-06-08T15:01:10.000Z

just for the record: other providers are dealing with this by having a separate "helm repository" per application, and not every application mangled together in one registry. so the index can be fairly smaller, and not everybody is syncing everything they even not need.

for example: https://kubernetes.github.io/ingress-nginx/index.yaml

vs: https://charts.bitnami.com/bitnami/index.yaml

But then, we would need to helm repo add bitnami-rabbit ... for each and every chart? That seems like an anti-pattern.

Answer 97 · 2022-06-08T15:17:13.000Z

just for the record: other providers are dealing with this by having a separate "helm repository" per application, and not every application mangled together in one registry. so the index can be fairly smaller, and not everybody is syncing everything they even not need.
for example: https://kubernetes.github.io/ingress-nginx/index.yaml
vs: https://charts.bitnami.com/bitnami/index.yaml

But then, we would need to helm repo add bitnami-rabbit ... for each and every chart? That seems like an anti-pattern.

I think generally speaking most people are only using a handful of charts?

It’s not much different than adding another chart/repo that isn’t maintained by bitnami?

Answer 98 · 2022-06-08T15:23:59.000Z

@mattb18 think about how things like KubeApps work (and also synaptic/apt, yum/dnf, and every other similar service). They're based around the idea that there is one repo with an index of all the packages.

Answer 99 · 2022-06-08T20:51:15.000Z

But then, we would need to helm repo add bitnami-rabbit ... for each and every chart? That seems like an anti-pattern.

This is quite common. If you look at any of the large community-driven chart aggregators, this is how they're built.

@mattb18 think about how things like KubeApps work (and also synaptic/apt, yum/dnf, and every other similar service). They're based around the idea that there is one repo with an index of all the packages.

They're absolutely not. They aggregate multiple repositories into a pool of applications. Take a look in /etc/apt/sources.list, or /etc/yum.repos.d/. Or indeed, more pertinent, helm was specifically designed to use multiple repos.