ConfigMap size limit prevents storing release
databus23 opened this issue · 26 comments
We hit a problem creating a release from a somewhat larger chart:
2016-10-19 09:42:22.348992 I | Updating "v1" (v1) in storage
2016-10-19 09:42:22.464534 I | configmaps: update: failed to update: ConfigMap "v1.v1" is invalid: data: Too long: must have at most 1048576 characters
2016-10-19 09:42:22.464583 I | warning: Failed to update release "v1": ConfigMap "v1.v1" is invalid: data: Too long: must have at most 1048576 characters
A helm install --dry-run --debug . > manifest produces a 600 KB file, base64 encoding that file gives a 810KB file.
It seems the serialised release somehow still gets larger then 1MB hitting the maximum size limit for a ConfigMap.
Some background: We are trying to install a complete openstack as a helm release and therefore have more then the usual handful of specs in the installed chart. The top-level chart package has a gzipped size of 113KB, unpacked its about 2MB.
I understand that this is pushing the limits of what helm is used for at the moment but in general I think the chart is not unreasonable in size.
One improvement to increase the size limit could be achieved by adding compression to the configmap storage backend. This would increase the practical size limit of release by a factor of 5 easily given that a release contains a lot of repetitive (compressible) data. Would a PR for that considered useful?
Any other ideas to fix our issue (apart from splitting up the release) ?
A PR for that would be very useful, especially if it is backwards compatible. I have no doubt that we'll see this issue again.
While the there are still limits to the chart/release size that can be stored in configmaps #1420 fixed the reported problem so I'm closing this.
Maybe this should be re-opened? #1420 enabled compression to delay the point at which this problem occurs but didn't address the fact that there is a relatively small limit. We're running into a similar issue. It seems Helm tries to store all the content of the chart's directory. We have cordoned off config for multiple environments, e.g. config/pd/, config/qa/, and each is only 300 KB but Helm fails to record the release due to the 1MB ConfigMap limit which is only reached by including all of config/ even though we are specifying specific values files. Maybe we should just keep them outside the chart directory?
$ helm install --kube-context pd -n infrastructure -f config/pd/values.yaml -f <(ansible-vault view config/pd/secrets.yaml.vault) --namespace infrastructure .
NAME: infrastructure
Error: getting deployed release "infrastructure": release: not found
$ kubectl logs --context=pd --namespace=kube-system tiller-deploy-3210613906-x3z3z | grep 'Failed to record release' | tail -n 1
2017/03/24 17:30:17 release_server.go:817: warning: Failed to record release "infrastructure": ConfigMap "infrastructure.v1" is invalid: data: Too long: must have at most 1048576 characters
$ kubectl delete namespace --context pd --cascade infrastructure
namespace "infrastructure" deleted
$ git rm -r config/qa
rm 'kube/helm/infrastructure/config/qa/secrets.yaml.vault'
rm 'kube/helm/infrastructure/config/qa/values.yaml'
$ git rm -r config/dr
rm 'kube/helm/infrastructure/config/dr/secrets.yaml.vault'
rm 'kube/helm/infrastructure/config/dr/values.yaml'
$ helm install --kube-context pd -n infrastructure -f config/pd/values.yaml -f <(ansible-vault view config/pd/secrets.yaml.vault) --namespace infrastructure .
NAME: infrastructure
...
STATUS: DEPLOYED
...
It looks like in our case we can use .helmignore because the --values are included in the release regardless.
This is giving us trouble as well. Is it not possible to break this file up into multiple config maps behind the scenes to get around the 1mb limit?
1MB does seem constraining in the 21st Century. Does the community agree that this limit should be lifted? What is it that makes it hard to lift this limit to something MUCH higher? I do think there is value in keeping Helm charts of modest size so lifting this limit should not be construed as encouragement to build ever larger Helm charts.
+1 to re-open this ticket
This limitation is a significant road block for helm to be used in broad spectrum of applications. Usually I package all the individual pieces ( front-end, back-end, database etc.) as a single helm chart to create a single command deployment for our application. I am having way to many issues hitting the limit of storing deployment configmap. In the example below I have only a single application WSO2 which consists of 6 individual pieces. Since I have to provide all the configuration files to all 6, the overall chart size is exceeding a limit of 1048576 characters. Helm is a great tool, but this limit must be addressed in order for helm to be used for all kind of deployments.
$ helm dependency update; helm install -n esb --set tags.dev-values=true .
Hang tight while we grab the latest from your chart repositories...
...Unable to get an update from the "local" chart repository (http://127.0.0.1:8879/charts):
Get http://127.0.0.1:8879/charts/index.yaml: dial tcp 127.0.0.1:8879: getsockopt: connection refused
...Successfully got an update from the "incubator" chart repository
...Successfully got an update from the "stable" chart repository
Update Complete. Happy Helming!
Saving 4 charts
Deleting outdated charts
NAME: esb
Error: getting deployed release "esb": release: "esb" not found
$ kubectl logs tiller-deploy-2543526833-fltvb --namespace kube-system
[storage] 2017/10/27 15:41:33 getting release history for "esb"
[storage] 2017/10/27 15:41:33 creating release "esb.v1"
[storage/driver] 2017/10/27 15:41:34 create: failed to create: ConfigMap "esb.v1" is invalid: data: Too long: must have at most 1048576 characters
[tiller] 2017/10/27 15:41:34 warning: Failed to record release esb: ConfigMap "esb.v1" is invalid: data: Too long: must have at most 1048576 characters
$ helm dependency update; helm install -n esb --set tags.dev-values=true . --dry-run --debug | base64 > /tmp/manifest ; ls -l /tmp/manifest
-rw-rw-r-- 1 robert robert 2314317 Oct 27 11:55 /tmp/manifest
This (1MB configmap limit) is not something that can be directly resolved in helm. The limitation is made upstream, so we can't really do anything to change that limitation.
The better approach would be to discuss alternative approaches in a design document as I described here so that when Helm 3 development begins, we can discuss alternative solutions such as what @jwomeara suggested.
This is the root cause of #1996
An alternative approach to storing the entire chart in a single ConfigMap should definitely be created so that Helm is not limited by this well known upstream constraint.
In addition to the storage size, because Secrets are being stored I would recommend that the Chart should not be stored within an "insecure" ConfigMap.
+1 to re-open this as well. Specifically it would be nice to have several options for back-end storage.
In the mean time, what have people been doing as a work-around? I'm trying to create 1 helm chart that has dependencies on all the other charts that make up the application (~20 charts). This puts it dangerously close to the 1MB limit. Is there a better way of organizing applications made up of many charts? Should i, for example, have each chart be it's own release? If so, how do i manage/automate creating a release for all the charts? Scripts?
Yeah, the limit is hurting us pretty bad. We use helm to deploy environment wide configs, these include different types of service tokens, which are pretty huge in size and don't compress well.
I think that if we could get some consensus on what a good alternative backend would be, we could get some people to write the driver. Drivers are actually pretty simple.
Two more cents here. I just ran into this problem. Until I found this issue, I didn't realize that helm was blindly putting all that is in the chart directory into the ConfigMap. Finding this helped me figure out that the temporary, somewhat large file in my chart directory was causing my issue.
Long story short, maybe helm should only be putting relevant, chart related dirs/files/info into the ConfigMap, not everything it sees in the chart directory.
@paulbsch thank you for your feedback. Unfortunately, "chart related dirs/files/info" is completely arbitrary, as they can be accessed during rendering time via the .Files object. Restricting the release ConfigMap to certain templates, known files and the like could potentially break existing use cases.
That being said, the Helm chart does have a concept of a .helmignore file, which allows users to specify which files to ignore during helm package. Will that help?
Yea. I know about .helmignore. I see what you're saying. I was just thinking that it might be reasonable to limit things to Chart.yaml, values.yaml, .yaml, charts/, templates/* ... For example.
Maybe some output/debug mode could print out what is being included in the ConfigMap? (maybe it does that already and I didn't see it?) That would have helped me in this case.
that would be a great option for helm package --debug or something similar. I agree.
Even just a check that lets a user know sooner would be nice. Something at the beginning of a helm install that prompts with something like: "Hey there, this won't work because your chart is over 1MB, and if you continue you will have to remove this deployment manually (which is a gigantic pain), are you sure you want to continue?"
I've proposed an extension to Kubernetes for this. Please take a look at the kubernetes issue #88709.
With this extension, you upload your configuration to s3 (or simply git) and reference it in a File resource. Finally the configmap references this file resource.
Feel free to upvote that proposal to raise awareness.
Current etcd version have 1.5mb limit already, with ability to increase it: https://etcd.io/docs/v3.5/dev-guide/limit/
By default, the maximum size of any request is 1.5 MiB. This limit is configurable through --max-request-bytes flag for etcd server.
Does this mean, that we can increase this limit in fresh k8s versions too?
/reopen Many years past, this is still not resolved, is there a way to split a chart to store in multi configmap to resolve this?
According to the docs, the sql backend can be used if the storage requirements exceed the size limit of etcd (e.g. configmaps or secrets):
https://helm.sh/docs/topics/advanced/#storage-backends
Using such a storage backend is particularly useful if your release information weighs more than 1MB (in which case, it can't be stored in ConfigMaps/Secrets because of internal limits in Kubernetes' underlying etcd key-value store).
I doubt there will ever be on option to split the the release across multiple secrets/configmaps.
I think this issue should be re-opened as it's blocker for the most of Cloud-managed platform (EKS, GKE, ACK), which limit secret/configmap size to 1MB (~ 1048576 bytes). Such popular charts as kube-prometheus-stack won't work. This is fundamental Helm design limitation, which should be addressed.
Sadly, this is still and issue for us. I guess it's nice to have the option to write the config to SQL, but I would prefer not to need an external dependency to get around this issue. If you're not willing to split the release across multiple configmaps, would you consider adding the ability to write the config to a file in a persistent volume instead? That way we wouldn't have to rely on an external dependency like SQL and could work within the confines of kubernetes.
There is unfortunately little to no enthousiasm for splitting releases across multiple configmaps although it is IMO the best option. I have a HIP PR helm/community#256 and a POC implementation PR #11087 to split release data >1MB into multiple configmaps.
I ran into this limitation, by using Bitnami Keycloak. I have 4 files to import, with a job. The ConfigMap is 1.3 MB, so Im stuck. Im now trying to work around this. First idea, was to create a CRD(that can hold up to 50MB, if Im correct). I did that, now Im trying to mount that config file into my Job and let it consume.
Proposal would be, to have some basic CRD "BigConfigMap"? That could possibly fix the problem, for most of the people.
I also ran into this limitation with multiple charts like the confluent kafka crds