catatonit keeps a mount directory open, preventing unmounting
Opened this issue · 6 comments
Debian bookworm (lite, headless) on a Raspberry Pi, using podman, although no containers are running.
We have a directory that is a cifs mount of an SMB file system (from a Mac). We could not unmount the directory because it was busy. lsof showed that a catatonit process/thread (related to podman, I believe) was holding the directory open.
I can imagine that catatonit is interested in mounted file systems, but, I would not expect catatonit to be so intrusive as to hold a file/directory open, thus prevent unmounting.
Regards
Does this happen with catatonit v0.1.6 and later? We added can de to close all fds >=3 a while ago (see #14 and #12) and so it seems unlikely that we would have any fds open as a result. Pre-0.1.6 it was possible for us to keep files open that were passed to the container process, which looks similar to what you're describing.
(FWIW, catatonit is a very simple program, it only opens /dev/tty
and possibly /proc/self/fd
and closes them before doing a signal passing loop.)
Also, if no containers are running there should be no catatonit process -- catatonit is a pid1 for containers.
$ catatonit --version
tini version 0.1.7_catatonit
It appears that podman
is using the -P
option of catatonit as per user.slice -> podman-pause-0db74da4.scope
:
$ systemctl status
● raspberrypi3a
State: running
Units: 305 loaded (incl. loaded aliases)
Jobs: 0 queued
Failed: 0 units
systemd: 252.17-1~deb12u1
CGroup: /
├─init.scope
│ └─1 /sbin/init
├─system.slice
snip
└─user.slice
└─user-1000.slice
├─session-1.scope
snip
└─user@1000.service
├─init.scope
│ ├─714 /lib/systemd/systemd --user
│ └─715 "(sd-pam)"
├─session.slice
│ └─dbus.service
│ └─841 /usr/bin/dbus-daemon --session --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
└─user.slice
└─podman-pause-0db74da4.scope
└─829 catatonit -P
Catatonit has several objects reported by lsof
, where /mnt/wd
is the mount that catatonit keeps busy even if no other processes are using it. And that mount is created after the podman pause service is running, suggesting to me that catatonit responds to some notification about the filesystem, and maybe sets its current working directory there...?
$ lsof | fgrep catatonit
COMMAND PID TID TASKCMD USER FD TYPE DEVICE SIZE/OFF NODE NAME
catatonit 829 pi cwd DIR 0,40 0 18989662 /mnt/wd
catatonit 829 pi rtd DIR 179,2 4096 2 /
catatonit 829 pi txt REG 179,2 600176 34185 /usr/bin/catatonit
catatonit 829 pi 0u CHR 1,3 0t0 5 /dev/null
catatonit 829 pi 1u CHR 1,3 0t0 5 /dev/null
catatonit 829 pi 2u CHR 1,3 0t0 5 /dev/null
catatonit 829 pi 3u a_inode 0,13 0 1060 [signalfd]
I said:
... where /mnt/wd is the mount that catatonit keeps busy even if no other processes are using it. And that mount is created after the podman pause service is running...
I'm no longer sure about that claim regarding the order of things.
What I have definitely just seen is that if the podman catatonit -P
starts after the cifs mount is already in place, the catatonit process has the cwd
entry for the mounted directory. That is, I don't have evidence that catatonit is actively responding to filesystem changes -- I do have evidence that when runs it does something special with the existing cifs mount...
Ah, the issue is that catatonit is being run inside the mount (by podman presumably). I guess we could change directory to /
but if the process is being run with the mount as the root of the mount namespace there's not much we can do. If anything this seems like a podman issue.
What does ls -ld /proc/$(pgrep catatonit)/root
output?
I'll have to check if podman is being used from the mount directory. That will require a reboot and there's a big build ongoing, so it will have to wait until tomorrow...
$ ls -ld /proc/$(pgrep catatonit)/root
lrwxrwxrwx 1 pi pi 0 Nov 4 02:09 /proc/1375/root -> /
I believe your suspicions are correct. It appears that the first time (in the user-specific systemd slice) that a user runs podman it creates a "hidden" container running catatonit -P
with cwd being the directory from which podman was run. Subsequent use of podman from other directories (or other simultaneous logins by the user) has no effect on the hidden container. This container does not show up in podman ps
, and it persists until the user-specific slice terminates.
I will file an issue with the podman project. It seems to me that the hidden container should have its cwd in the user's home directory, or some suitable /var
area.