Systemd’s CPU affinity not honored when LXD snap is installed in Ubuntu 22.04.

Question

Systemd’s CPU affinity not honored when LXD snap is installed in Ubuntu 22.04.

Closed this issue 5 months ago · 4 comments

Required information

Distribution: Ubuntu
Distribution version: 22.04
The output of "snap list --all lxd core20 core22 core24 snapd":

ubuntu@test:~$ snap list --all lxd core20 core22 core24 snapd
Name    Version        Rev    Tracking       Publisher   Notes
core20  20240111       2182   latest/stable  canonical✓  base
lxd     5.0.3-9a1d904  27428  5.0/stable/…   canonical✓  -
snapd   2.61.2         21184  latest/stable  canonical✓  snapd

The output of "lxc info" or if that fails:
- Kernel version: 5.15.0-101-generic
- LXC version: 5.0.3
- LXD version: 5.0.3-9a1d904
- Storage backend in use: default

Issue description

When systemd’s CPUaffinity is used in Ubuntu 22.04, it is not enforced if the LXD snap version <6.1 is installed.

Steps to reproduce

get a host (or VM) with lxd snap installed and systemd's CPUaffinity configured to bound systemd and its childs to some CPUs, e.g., CPUs=0-1:

uvt-kvm create test release=jammy arch=amd64 --memory 4096 --cpu 4 --disk 50  
uvt-kvm ssh test

Within the instance:

ubuntu@test:~$ echo "CPUAffinity=0-1" | sudo tee -a /etc/systemd/system.conf
ubuntu@test:~$ sudo systemctl daemon-reexec

ubuntu@test:~$ grep -i affinity /etc/systemd/system.conf
CPUAffinity=0-1
ubuntu@test:~$ sudo taskset -cp 1
pid 1's current affinity list: 0-3

since we have CPUAffinity=0-1 in /etc/systemd/system.conf, the expectation would be to get pid 1's current affinity list: 0-1 from taskset, but this is not honored and any CPU (machine below has 4 cores, so CPU=0-3) is allowed.

Just as a test of persistence for the issue, reboot the system and still no affinity set:

ubuntu@test:~$ sudo reboot
...
ubuntu@test:~$ sudo taskset -cp 1
pid 1's current affinity list: 0-3

Remove the lxd snap then the CPUaffinity starts to get enforced:

ubuntu@test:~$ snap list
Name Version Rev Tracking Publisher Notes
core20 20240111 2182 latest/stable canonical✓ base
lxd 5.0.3-9a1d904 27428 5.0/stable/… canonical✓ -
snapd 2.61.2 21184 latest/stable canonical✓ snapd
ubuntu@test:~$ sudo snap remove lxd
lxd removed
ubuntu@test:~$ grep -i affinity /etc/systemd/system.conf
CPUAffinity=0-1
ubuntu@test:~$ sudo taskset -cp 1
pid 1's current affinity list: 0,1

Notice above that after the lxd snap is removed then pid 1's current affinity list: 0,1 meaning that systemd and its childs are bound to CPU=0-1

Answer 1 · 2024-11-08T09:47:32.000Z

@mihalicyn want to investugate this one?

Answer 2 · 2024-11-08T09:57:40.000Z

Same issue in Ubuntu Focal with lxd 4.0.9-a29c6f1:

ubuntu@test:~$ cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=20.04
DISTRIB_CODENAME=focal
DISTRIB_DESCRIPTION="Ubuntu 20.04.5 LTS"

ubuntu@test:~$ uname -r
5.4.0-139-generic


ubuntu@test:~$ echo "CPUAffinity=0-1" | sudo tee -a /etc/systemd/system.conf
CPUAffinity=0-1
ubuntu@test:~$ grep -i affi /etc/systemd/system.conf
#CPUAffinity=1 2
CPUAffinity=0-1
ubuntu@test:~$ sudo systemctl daemon-reexec

# affinity not enforced, see below
ubuntu@test:~$ sudo taskset -cp 1
pid 1's current affinity list: 0-3

# removed the LXD snap, and affinity was then enforced
ubuntu@test:~$ snap list
Name    Version        Rev    Tracking       Publisher   Notes
core20  20230126       1822   latest/stable  canonical✓  base
lxd     4.0.9-a29c6f1  24061  4.0/stable/…   canonical✓  -
snapd   2.58           17950  latest/stable  canonical✓  snapd
ubuntu@test:~$ sudo snap remove lxd
lxd removed
ubuntu@test:~$ sudo taskset -cp 1
pid 1's current affinity list: 0,1

Answer 3 · 2024-11-08T12:10:38.000Z

This is quite and odd issue and may not necessarily be a problem with the lxd:

ubuntu@test:~$ snap list
Name    Version        Rev    Tracking       Publisher   Notes
core20  20240111       2182   latest/stable  canonical✓  base
lxd     5.0.3-9a1d904  27428  5.0/stable/…   canonical✓  -
snapd   2.61.2         21184  latest/stable  canonical✓  snapd
ubuntu@test:~$ echo "CPUAffinity=0-1" | sudo tee -a /etc/systemd/system.conf; sudo systemctl daemon-reexec
CPUAffinity=0-1
ubuntu@test:~$ grep -i affinity /etc/systemd/system.conf
#CPUAffinity=
CPUAffinity=0-1
ubuntu@test:~$ sudo taskset -cp 1
pid 1's current affinity list: 0-3
ubuntu@test:~$ snap list
Name    Version        Rev    Tracking       Publisher   Notes
core20  20240111       2182   latest/stable  canonical✓  base
lxd     5.0.3-9a1d904  27428  5.0/stable/…   canonical✓  -
snapd   2.61.2         21184  latest/stable  canonical✓  snapd
ubuntu@test:~$ sudo snap refresh core20 --channel latest/stable --revision c
core20 20240416 from Canonical✓ refreshed
ubuntu@test:~$ snap list
Name    Version        Rev    Tracking       Publisher   Notes
core20  20240416       2318   latest/stable  canonical✓  base
lxd     5.0.3-9a1d904  27428  5.0/stable/…   canonical✓  -
snapd   2.61.2         21184  latest/stable  canonical✓  snapd
ubuntu@test:~$ sudo taskset -cp 1
pid 1's current affinity list: 0,1

From above, notice that as soon as I refreshed core20 the affinity started being enforced. Now, the odd thing about this is that I refreshed core20 to revision 2318 which I have nodes running and the issue is present there.

So, in short, unclear where the problem is and to which of these 3 snaps is related (if any).

Answer 4 · 2024-11-11T21:36:51.000Z

closing as this does not seem related to LXD or any snap for that matter.