Wrong /dev/null permission making ubuntu jammy deployment impossible
Closed this issue · 6 comments
When switching the sandbox project to deploy ubuntu jammy, running apt update
with the cexec
action fails due to not having permission to write to /dev/null
Expected Behaviour
Being able to run apt update
when deploying the ubuntu jammy image, permissions on /dev/null
need to be 666
for apt update
to work.
Current Behaviour
apt update
in cexec
action fails when deploying the ubuntu jammy image because it can't write to /dev/null
, permissions on /dev/null
are 660
and apt update
doesn't work
Possible Solution
First I updated cexec
container to mount /dev
as rw
so I could update the permissions from the template. Then I switched to a more general approach where I updated hook-docker
to set the correct permissions:
hook-docker/main.go
────────────────────────────────────────────────────────────────────────────────────────────────────────────
──────────────────┐
31: func main() { │
──────────────────┘
31 ⋮ 31 │ fmt.Println("Starting Tink-Docker")
32 ⋮ 32 │ go rebootWatch()
33 ⋮ 33 │
⋮ 34 │ fmt.Println("Make /dev/null writeable for all users!")
⋮ 35 │ cmd := exec.Command("chmod", "666", "/dev/null")
⋮ 36 │ cmd.Stdout = os.Stdout
⋮ 37 │ cmd.Stderr = os.Stderr
⋮ 38 │ err := cmd.Run()
⋮ 39 │ if err != nil {
⋮ 40 │ panic(err)
⋮ 41 │ }
⋮ 42 │
34 ⋮ 43 │ // Parse the cmdline in order to find the urls for the repository and path to the cert
35 ⋮ 44 │ content, err := ioutil.ReadFile("/proc/cmdline")
36 ⋮ 45 │ if err != nil {
──────────────────┐
74: func main() { │
──────────────────┘
65 ⋮ 74 │ }
66 ⋮ 75 │
67 ⋮ 76 │ // Build the command, and execute
68 ⋮ │ cmd := exec.Command("/usr/local/bin/docker-init", "/usr/local/bin/dockerd")
⋮ 77 │ cmd = exec.Command("/usr/local/bin/docker-init", "/usr/local/bin/dockerd")
69 ⋮ 78 │ cmd.Stdout = os.Stdout
70 ⋮ 79 │ cmd.Stderr = os.Stderr
71 ⋮ 80 │ err = cmd.Run()
While I got it working I don't know if there are better ways to solve this problem.
Steps to Reproduce (for bugs)
- Try deploying ubuntu jammy image with the sandbox
Context
Your Environment
-
Operating System and version (e.g. Linux, Windows, MacOS): Linux
-
How are you running Tinkerbell? Using Vagrant & VirtualBox, Vagrant & Libvirt, on Packet using Terraform, or give details: Sandbox & docker-compose deploying on bare-metal
-
Link to your project or a code example to reproduce issue:
Hey @lanquarden, thanks for reporting this. I definitely agree that /dev/null
should have 666 permissions. Weird thing is that I am able to reproduce this on one machine, but on another machine I cannot. i.e. installing the machine1
with Ubuntu Jammy works successfully.
In the successful install, /dev/null
has 660 permissions. So, it makes me think that there's possibly an environmental issue external to vagrant that is causing issues. I'm definitely not certain though. I will continue to investigate and update here.
Maybe the issue resides in apt update
, these commands are invoked with root user so it shouldn't complain about the 660 permission, unless apt is running some parts as a different user... Maybe this behavior is linked to the environment somehow. I didn't explore that route as I had found a workaround.
@jacobweinstock we run into the same with EKS Anywhere (we opened an AWS case as well as we have EKS-A Subscription).
In fact, @lanquarden is right and by default apt
is using the _apt
user for calling out to apt-key
(and to many other "sandboxed" tasks like downloading a package etc.) that would write to /dev/null
.
The main issue here is that with the current hook kernel, when you mount /dev
with devtmpfs
/dev/null
gets created with 0660 (and this happens inside the cexec container as well thanks to this):
While using Ubuntu 22.04 for example it gets 0666:
root@kls107:~# mkdir /mnt/dev
root@kls107:~# mount -r -t devtmpfs none /mnt/dev
root@kls107:~# ll /mnt/dev/null
crw-rw-rw- 1 root root 1, 3 Feb 6 09:29 /mnt/dev/null
root@kls107:~# mount|grep /mnt/dev
none on /mnt/dev type devtmpfs (ro,relatime,size=65716676k,nr_inodes=16429169,mode=755,inode64)
root@kls107:~#
We would love to get a fix (or any applicable workaround that does not require custom build cexec container/hook os) into EKS Anywhere.
I've sent a PR to cexec that implements a "harmless workaround" and allows the call of apt update in the following way:
CMD_LINE: echo 'nameserver IPOFYOURNAMESERVER' > /etc/resolv.conf && export NEEDRESTART_SUSPEND=true && apt -y update && apt install -y nfs-common open-iscsi....
Hey @lanquarden and @Cajga , this PR should resolve this. #200 . Once its landed, or before if you want to build from my branch, would you mind testing to validate its working for you both?
@jacobweinstock , thank you for the quick fix.
I can confirm that using the hook image fixed the /dev/*
permissions inside hook. I can also confirm that the /dev/null
has 0666
permissions inside the chrooted cexec action ( this surprised me as it is mounting devtmpfs (that had permission issues) instead of bind mounting /dev
from hook).