Nodes regularly failing after a day
Closed this issue · 6 comments
Hi there,
I understand this is not a bug with your implementation.
We are seeing an issue with nodes failing to start pods with the following errors.
MountVolume.SetUp failed for volume "default-token-hrpk9" : mount failed: exit status 1 Mounting command: systemd-run Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/f570149c-5462-11e8-a72b-02a9a02d1274/volumes/kubernetes.io~secret/default-token-hrpk9 --scope -- mount -t tmpfs tmpfs /var/lib/kubelet/pods/f570149c-5462-11e8-a72b-02a9a02d1274/volumes/kubernetes.io~secret/default-token-hrpk9 Output: Failed to start transient scope unit: Connection timed out
SystemD has become unresponsive.
We have a few CronJobs running which I suspect is causing the systemd to eventually be unable to mount new secrets. Increasing the CronJob rate hasn't made much affect.
The closest issue I have seen is kubernetes/kubernetes#57345 but the issue varies slightly. Remoting into the machine systemd is completely unresponsive.
systemctl list-units --all | wc -l
Failed to list units: Connection timed out
0
What I know
- System reboot fixes the issue.
- Systemd is an older version, and unlikely to be updated anytime soon
- Possibly not a kubernetes issue
Any suggestions on where to hunt?
Sorry, I have never seen this problem. So I'm afraid I cannot help with this :-(. In the past I saw lot of strange issues when the disk space run out. But I'm not sure if this is your case.
Thanks Jakub, I appreciate the quick response.
Just for the record - got the same issue with managed k8s on GCP and also because of high use of CronJobs, I guess the only solution is to get Systemd of version 237 which has a fix
I encountered such problem, reboot your system is the fast way to fix this case
same case here on Azure AKS - restart of all nodes helped
Same problem on GKE with COS nodes (systemd 232). Switching to Ubuntu nodes with systemd 237 to see if it solves the problems.