coreos/go-systemd

cpuset disappears from /sys/fs/cgroup/cgroup.subtree_control after adding it

Closed this issue · 3 comments

harche commented

After the OS boots and the kubelet starts, it adds cpuset to /sys/fs/cgroup/cgroup.subtree_control. However, it seems that kubelet calls the runc method setUnitProperties, which in turn calls SetUnitPropertiesContext from the go-systemd library. At this point, we notice that the cpuset, which was already present, gets removed (for reasons unknown). This causes kubelet to restart, but this time, the cpuset gets added back and remains in /sys/fs/cgroup/cgroup.subtree_control.

More details are in the description of kubernetes/kubernetes#122955

On my local OCP node, I was able to observe this issue as well,

[root@ip-10-0-2-54 ~]# systemctl restart kubelet
[root@ip-10-0-2-54 ~]# cat /sys/fs/cgroup/cgroup.subtree_control
cpu io memory hugetlb pids rdma misc
[root@ip-10-0-2-54 ~]# cat /sys/fs/cgroup/cgroup.subtree_control
cpu io memory hugetlb pids rdma misc
[root@ip-10-0-2-54 ~]# cat /sys/fs/cgroup/cgroup.subtree_control
cpu io memory hugetlb pids rdma misc
[root@ip-10-0-2-54 ~]# cat /sys/fs/cgroup/cgroup.subtree_control
cpu io memory hugetlb pids rdma misc
[root@ip-10-0-2-54 ~]# cat /sys/fs/cgroup/cgroup.subtree_control
cpu io memory hugetlb pids rdma misc
[root@ip-10-0-2-54 ~]# cat /sys/fs/cgroup/cgroup.subtree_control
cpu io memory hugetlb pids rdma misc
[root@ip-10-0-2-54 ~]# cat /sys/fs/cgroup/cgroup.subtree_control
cpu io memory hugetlb pids rdma misc
[root@ip-10-0-2-54 ~]# cat /sys/fs/cgroup/cgroup.subtree_control
cpu io memory hugetlb pids rdma misc
[root@ip-10-0-2-54 ~]# cat /sys/fs/cgroup/cgroup.subtree_control
cpu io memory hugetlb pids rdma misc
[root@ip-10-0-2-54 ~]# cat /sys/fs/cgroup/cgroup.subtree_control
cpu io memory hugetlb pids rdma misc
[root@ip-10-0-2-54 ~]# cat /sys/fs/cgroup/cgroup.subtree_control
cpuset cpu io memory hugetlb pids rdma misc                      <<<<<< cpuset is added
[root@ip-10-0-2-54 ~]# cat /sys/fs/cgroup/cgroup.subtree_control    
cpu io memory hugetlb pids rdma misc                             <<<<<< gone here. 
[root@ip-10-0-2-54 ~]# cat /sys/fs/cgroup/cgroup.subtree_control
cpuset cpu io memory hugetlb pids rdma misc                      <<<<<< kubelet restart, and it is back again. 
[root@ip-10-0-2-54 ~]# cat /sys/fs/cgroup/cgroup.subtree_control
cpuset cpu io memory hugetlb pids rdma misc
[root@ip-10-0-2-54 ~]# cat /sys/fs/cgroup/cgroup.subtree_control
cpuset cpu io memory hugetlb pids rdma misc
[root@ip-10-0-2-54 ~]# cat /sys/fs/cgroup/cgroup.subtree_control
cpuset cpu io memory hugetlb pids rdma misc
[root@ip-10-0-2-54 ~]# cat /sys/fs/cgroup/cgroup.subtree_control
cpuset cpu io memory hugetlb pids rdma misc
harche commented

I have observed this happening only on cgroup v2 systems.

harche commented

/close as it seems the bug was in the kubelet code.