kubernetes/kops

ulimit changed in pods between kops 1.28 -> master

Closed this issue · 2 comments

zetaab commented

/kind bug

1. What kops version are you running? The command kops version, will display
this information.

master

2. What Kubernetes version are you running? kubectl version will print the
version if a cluster is running or provide the Kubernetes version specified as
a kops flag.

1.29.1

3. What cloud provider are you using?

openstack & aws

4. What commands did you run? What is the simplest way to reproduce this issue?

old instance:

% kubectl exec -it node-exporter-95x5c -- sh
/ $ ulimit -n
1048576

new instance (with kops master):

% kubectl exec -it node-exporter-mc89j -- sh
/ $ ulimit -n
1024

5. What happened after the commands executed?

We can clearly see that something is setting new default ulimit only to 1024.

6. What did you expect to happen?

I expect that kOps does not change the behaviour between versions

zetaab commented

seems that /lib/systemd/system/containerd.service file contains in old instance LimitNOFILE=infinity but it does not exist anymore in kOps master.

zetaab commented

imo it is quite radical just remove that setting in #16151 without providing way to configure that and migrate projects to new behaviour. In slack conversation it was told that kops supports containerd nri already https://kops.sigs.k8s.io/cluster_spec/#nri-configuration and its possible to somehow configure ulimits using that. However, we should have somekind of window to migrate from old to new one, instead of just removing the old solution.

  1. we should mark this ulimit as deprecated and provide instructions how to migrate new way of doing
  2. delete this old ulimit behaviour in next release
  3. with boolean flag we could still allow the old behaviour longer.

This change is quite big. We have currently issues running databases in production because ulimit is limited to 1024. I have now done hotfix (own compiled version) to revert this ulimit change, but still we should have migration window for projects. Most of them are developers who are not interested of changing their deployment manifests just because something changes in platform itself. Also it might be problem for like third party helm charts, if those does not support specifying annotations for pods.