kinvolk/kube-spawn

kube-proxy stuck in CrashLoopBackOff

Closed this issue · 9 comments

robertgzr commented

kubelet logs:

kuberuntime_manager.go:458] Container {Name:kube-proxy Image:10.22.0.1:5000/hyperkube-amd64 Command:[/usr/local/bin/kube-proxy --kubeconfig=/var/lib/kube-proxy/kubeconfig.conf] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:kube-proxy ReadOnly:false MountPath:/var/lib/kube-proxy SubPath:} {Name:kube-proxy-token-bgzp8 ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath:}] LivenessProbe:nil ReadinessProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:&SecurityContext{Capabilities:nil,Privileged:*true,SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,ReadOnlyRootFilesystem:nil,} Stdin:false StdinOnce:false TTY:false} is dead, but RestartPolicy says that we should restart it

kube-proxy logs:

0410 15:56:52.119847       1 server.go:227] Using iptables Proxier.
W0410 15:56:52.148628       1 proxier.go:313] clusterCIDR not specified, unable to distinguish between internal and external traffic
I0410 15:56:52.148666       1 server.go:252] Tearing down userspace rules.
I0410 15:56:52.176541       1 conntrack.go:81] Set sysctl 'net/netfilter/nf_conntrack_max' to 131072
I0410 15:56:52.177035       1 conntrack.go:66] Setting conntrack hashsize to 32768
Error: write /sys/module/nf_conntrack/parameters/hashsize: operation not supported
Error: write /sys/module/nf_conntrack/parameters/hashsize: operation not supported
alban commented

According to the Linux kernel documentation here, we cannot write on /sys/module/nf_conntrack/parameters/hashsize if we are not in the initial network namespace and that would return -EOPNOTSUPP. This seems to be what happens there.

kube-proxy might run in a Docker container with --net=host (so it does not create a new network namespace), but if it is in a systemd-nspawn container with --private-network or similar network option, then it will not be in the initial network namespace.

What's the current value of your /sys/module/nf_conntrack/parameters/hashsize on the host? If it is already bigger than 32768, you have a workaround: patch kube-proxy to read the current value and ignore the error when it is already big enough.

  • 👍1
robertgzr commented

Yes we are running it inside a network namespace created by our cnispawn binary.
It's at 65536 on my host machine. So I guess we have to patch kube-proxy then.

alban commented

Another possible workaround: use --conntrack-max=0 according to kubernetes/kubernetes#24027 (comment).

robertgzr commented

I found that too. And this (which is kind of like our use case):
kubernetes/kubeadm#17 (comment)

We currently do this to fix that but it seems to fail. I get an error about validation when I run the same line

error: error validating "STDIN": error validating data: items[0].apiVersion not set; if you choose to ignore these errors, turn validation off with --validate=false

and when I turn validation off:

error: unable to recognize /v1, Kind=: no matches for /, Kind=
robertgzr commented

I can get it to work by adding the flag here:
https://github.com/kubernetes/kubernetes/blob/master/cmd/kubeadm/app/phases/addons/manifests.go#L71
and compiling kubeadm with that.

There seem to be plans to incorporate the ability to change kube-proxy flags via the kubeadm API
kubernetes/kubernetes#34522 but that is not possible yet.

robertgzr commented

@alban It works after patching kube-proxy here
https://github.com/thockin/kubernetes/blob/da0ac31182ffeebde65f72e09bc12e1af20dd73a/cmd/kube-proxy/app/server.go#L338
with a check if nf_conntrack_max is already high enough.

daveamit commented

For those who came here recently, --conntrack-max is now ( at time of v1.19.3) --conntrack-max-per-core

All you need to do it edti /var/snap/microk8s/current/args/kube-proxy and add --conntrack-max-per-core=0 at the end.