Quadlet failing to pull
Opened this issue · 10 comments
chsasank commented
Issue Description
Podman service file somehow timing out before pulling.
Steps to reproduce the issue
create the following .container file
$ cat /home/sasank/.config/containers/systemd/matrix-postgres.container
[Container]
Image=docker.io/postgres:12-alpine
Pod=matrix.pod
EnvironmentFile=/home/sasank/.johnny/matrix/postgres.env
PodmanArgs=
Exec=
Volume=/home/sasank/.johnny/matrix/postgres-data:/var/lib/postgresql/data
[Service]
Restart=always
[Install]
WantedBy=default.target
Reload daemon
$ systemctl --user daemon-reload
Start it
$ systemctl --user start matrix-postgres.service
Job for matrix-postgres.service canceled.
Describe the results you received
Look for status
$ systemctl --user status matrix-postgres.service
● matrix-postgres.service
Loaded: loaded (/home/sasank/.config/containers/systemd/matrix-postgres.container; generated)
Active: activating (start) since Wed 2024-10-09 18:02:02 IST; 324ms ago
Main PID: 3618748 (podman)
Tasks: 12 (limit: 38034)
Memory: 13.2M
CPU: 24ms
CGroup: /user.slice/user-1002.slice/user@1002.service/app.slice/matrix-postgres.service
└─3618748 /usr/bin/podman run --name=systemd-matrix-postgres --cidfile=/run/user/1002/matrix-postgres.cid --replace --rm --cgroups=split --sdnotify=conmon -d -v /home/sasank/.johnny/matrix/postg>
Oct 09 18:02:02 JOHNAIC systemd[904]: Starting matrix-postgres.service...
Oct 09 18:02:02 JOHNAIC matrix-postgres[3618748]: Trying to pull docker.io/library/postgres:12-alpine...
Oct 09 18:02:02 JOHNAIC matrix-postgres[3618748]: Pulling image //postgres:12-alpine inside systemd: setting pull timeout to 5m0s
Oct 09 18:02:03 JOHNAIC systemd[904]: Stopped matrix-postgres.service.
Describe the results you expected
Expected the image to pull and run
podman info output
host:
arch: amd64
buildahVersion: 1.37.4
cgroupControllers:
- memory
- pids
cgroupManager: systemd
cgroupVersion: v2
conmon:
package: conmon_100:2.1.12-1_amd64
path: /usr/bin/conmon
version: 'conmon version 2.1.12, commit: e21e7c85b7637e622f21c57675bf1154fc8b1866'
cpuUtilization:
idlePercent: 94.15
systemPercent: 1.57
userPercent: 4.27
cpus: 12
databaseBackend: boltdb
distribution:
codename: jammy
distribution: neon
version: "22.04"
eventLogger: journald
freeLocks: 2024
hostname: JOHNAIC
idMappings:
gidmap:- container_id: 0
host_id: 1002
size: 1 - container_id: 1
host_id: 231072
size: 65536
uidmap: - container_id: 0
host_id: 1002
size: 1 - container_id: 1
host_id: 231072
size: 65536
kernel: 6.5.0-45-generic
linkmode: dynamic
logDriver: journald
memFree: 3304464384
memTotal: 33372499968
networkBackend: netavark
networkBackendInfo:
backend: netavark
dns:
package: aardvark-dns_1.6.0-0ubuntu22.04+obs34.25_amd64
path: /usr/libexec/podman/aardvark-dns
version: aardvark-dns 1.6.0
package: netavark_1.3.0-0ubuntu22.04+obs22.9_amd64
path: /usr/libexec/podman/netavark
version: netavark 1.3.0
ociRuntime:
name: crun
package: crun_101:1.14.4-0ubuntu22.04+obs70.24_amd64
path: /usr/bin/crun
version: |-
crun version 1.14.4
commit: a220ca661ce078f2c37b38c92e66cf66c012d9c1
rundir: /run/user/1002/crun
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
os: linux
pasta:
executable: /usr/bin/pasta
package: passt_100:0.0+20240906.6b38f072-1_amd64
version: |
pasta 0.0+20240906.6b38f072
Copyright Red Hat
GNU General Public License, version 2 or later
https://www.gnu.org/licenses/old-licenses/gpl-2.0.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
remoteSocket:
exists: false
path: /run/user/1002/podman/podman.sock
rootlessNetworkCmd: pasta
security:
apparmorEnabled: false
capabilities: CAP_AUDIT_WRITE,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_MKNOD,CAP_NET_BIND_SERVICE,CAP_NET_RAW,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
rootless: true
seccompEnabled: true
seccompProfilePath: /usr/share/containers/seccomp.json
selinuxEnabled: false
serviceIsRemote: false
slirp4netns:
executable: /usr/bin/slirp4netns
package: slirp4netns_100:1.3.1-1_amd64
version: |-
slirp4netns version 1.3.1
commit: unknown
libslirp: 4.6.1
SLIRP_CONFIG_VERSION_MAX: 5
libseccomp: 2.5.3
swapFree: 30199177216
swapTotal: 36709785600
uptime: 174h 6m 15.00s (Approximately 7.25 days)
variant: ""
plugins:
authorization: null
log:
- container_id: 0
- k8s-file
- none
- passthrough
- journald
network: - bridge
- macvlan
- ipvlan
volume: - local
registries:
search: - docker.io
store:
configFile: /home/sasank/.config/containers/storage.conf
containerStore:
number: 12
paused: 0
running: 11
stopped: 1
graphDriverName: overlay
graphOptions: {}
graphRoot: /home/sasank/.local/share/containers/storage
graphRootAllocated: 463303442432
graphRootUsed: 149483036672
graphStatus:
Backing Filesystem: extfs
Native Overlay Diff: "true"
Supports d_type: "true"
Supports shifting: "false"
Supports volatile: "true"
Using metacopy: "false"
imageCopyTmpDir: /var/tmp
imageStore:
number: 26
runRoot: /run/user/1002/containers
transientStore: false
volumePath: /home/sasank/.local/share/containers/storage/volumes
version:
APIVersion: 5.2.4
Built: 0
BuiltTime: Thu Jan 1 05:30:00 1970
GitCommit: ""
GoVersion: go1.23.2
Os: linux
OsArch: linux/amd64
Version: 5.2.4
Podman in a container
No
Privileged Or Rootless
Rootless
Upstream Latest Release
No
Additional environment details
No response
Additional information
No response
chsasank commented
If I pull the image manually, it works as expected
$ podman pull docker.io/library/postgres:12-alpine
$ systemctl --user start matrix-postgres.service
$ systemctl --user status matrix-postgres.service
● matrix-postgres.service
Loaded: loaded (/home/sasank/.config/containers/systemd/matrix-postgres.container; generated)
Active: active (running) since Wed 2024-10-09 18:07:30 IST; 3s ago
Main PID: 3623990 (conmon)
Tasks: 8 (limit: 38034)
Memory: 19.1M
CPU: 98ms
CGroup: /user.slice/user-1002.slice/user@1002.service/app.slice/matrix-postgres.service
├─libpod-payload-c5a823b759653d471854b424856674ad7fc3342a94ba05c72afc8f1f0a31b770
│ ├─3623992 postgres
│ ├─3624031 "postgres: checkpointer "
│ ├─3624032 "postgres: background writer "
│ ├─3624033 "postgres: walwriter "
│ ├─3624034 "postgres: autovacuum launcher "
│ ├─3624035 "postgres: stats collector "
│ └─3624036 "postgres: logical replication launcher "
└─runtime
└─3623990 /usr/bin/conmon --api-version 1 -c c5a823b759653d471854b424856674ad7fc3342a94ba05c72afc8f1f0a31b770 -u c5a823b759653d471854b424856674ad7fc3342a94ba05c72afc8f1f0a31b770 -r /usr/bin/cr>
Oct 09 18:07:30 JOHNAIC matrix-postgres[3623951]: c5a823b759653d471854b424856674ad7fc3342a94ba05c72afc8f1f0a31b770
Oct 09 18:07:30 JOHNAIC systemd-matrix-postgres[3623990]:
Oct 09 18:07:30 JOHNAIC systemd-matrix-postgres[3623990]: PostgreSQL Database directory appears to contain a database; Skipping initialization
Oct 09 18:07:30 JOHNAIC systemd-matrix-postgres[3623990]:
Oct 09 18:07:30 JOHNAIC systemd-matrix-postgres[3623990]: 2024-10-09 12:37:30.318 UTC [1] LOG: starting PostgreSQL 12.20 on x86_64-pc-linux-musl, compiled by gcc (Alpine 13.2.1_git20240309) 13.2.1 20240309,>
Oct 09 18:07:30 JOHNAIC systemd-matrix-postgres[3623990]: 2024-10-09 12:37:30.318 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
Oct 09 18:07:30 JOHNAIC systemd-matrix-postgres[3623990]: 2024-10-09 12:37:30.318 UTC [1] LOG: listening on IPv6 address "::", port 5432
Oct 09 18:07:30 JOHNAIC systemd-matrix-postgres[3623990]: 2024-10-09 12:37:30.322 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
Oct 09 18:07:30 JOHNAIC systemd-matrix-postgres[3623990]: 2024-10-09 12:37:30.340 UTC [22] LOG: database system was shut down at 2024-10-09 12:18:54 UTC
Oct 09 18:07:30 JOHNAIC systemd-matrix-postgres[3623990]: 2024-10-09 12:37:30.355 UTC [1] LOG: database system is ready to accept connections
vrothberg commented
How long does it take to pull the image? It looks like you're running into a timeout that you can extend if needed.
chsasank commented
When I pulled the image manually it hardly takes twenty seconds.
$ podman image rm docker.io/postgres:12-alpine
Untagged: docker.io/library/postgres:12-alpine
Deleted: 994c5880d6731b241afaae87c0b25ab2f005785f4175ef147ebcd3f0c652343c
$ podman image prune
WARNING! This command removes all dangling images.
Are you sure you want to continue? [y/N] y
$ time podman image pull docker.io/postgres:12-alpine
Trying to pull docker.io/library/postgres:12-alpine...
Getting image source signatures
Copying blob fb60b7546f99 done |
Copying blob e7368e03b632 done |
Copying blob 43c4264eed91 skipped: already exists
Copying blob 738c839a1612 done |
Copying blob a4a6aafd4b69 done |
Copying blob 038cdbd7acd1 done |
Copying blob e8698ad92e74 done |
Copying blob 3a291eff5ef1 done |
Copying blob 4a28cf0127a7 done |
Copying blob e9be2a968901 done |
Copying blob eb78f2fdb93e done |
Copying config 994c5880d6 done |
Writing manifest to image destination
994c5880d6731b241afaae87c0b25ab2f005785f4175ef147ebcd3f0c652343c
real 0m11.857s
user 0m3.106s
sys 0m0.956s
chsasank commented
Can reproduce the issue by simply removing the image
$ podman image rm docker.io/postgres:12-alpine
Untagged: docker.io/library/postgres:12-alpine
Deleted: 994c5880d6731b241afaae87c0b25ab2f005785f4175ef147ebcd3f0c652343c
$ systemctl --user stop matrix-postgres.service
$ systemctl --user start matrix-postgres.service
Job for matrix-postgres.service canceled.
$ systemctl --user status matrix-postgres.service
○ matrix-postgres.service
Loaded: loaded (/home/sasank/.config/containers/systemd/matrix-postgres.container; generated)
Active: inactive (dead) since Wed 2024-10-09 18:13:07 IST; 2s ago
Process: 3630984 ExecStart=/usr/bin/podman run --name=systemd-matrix-postgres --cidfile=/run/user/1002/matrix-postgres.cid --replace --rm --cgroups=split --sdnotify=conmon -d -v /home/sasank/.johnny/matr>
Process: 3631096 ExecStopPost=/usr/bin/podman rm -v -f -i --cidfile=/run/user/1002/matrix-postgres.cid (code=exited, status=0/SUCCESS)
Main PID: 3630984 (code=exited, status=0/SUCCESS)
CPU: 71ms
Oct 09 18:13:07 JOHNAIC systemd[904]: Starting matrix-postgres.service...
Oct 09 18:13:07 JOHNAIC matrix-postgres[3630984]: Trying to pull docker.io/library/postgres:12-alpine...
Oct 09 18:13:07 JOHNAIC matrix-postgres[3630984]: Pulling image //postgres:12-alpine inside systemd: setting pull timeout to 5m0s
Oct 09 18:13:07 JOHNAIC systemd[904]: Stopped matrix-postgres.service.
chsasank commented
Surprisingly I can not reproduce this for a different container
$ cat /home/sasank/.config/containers/systemd/matrix-synapse.container
[Container]
Image=docker.io/matrixdotorg/synapse:v1.116.0
Pod=matrix.pod
EnvironmentFile=/home/sasank/.johnny/matrix/synapse.env
PodmanArgs=
Exec=
Volume=/home/sasank/.johnny/matrix/matrix-data:/data
[Service]
Restart=always
[Install]
WantedBy=default.target
$ podman image rm docker.io/matrixdotorg/synapse:v1.116.0
$ time systemctl --user start matrix-synapse.service
real 0m12.101s
user 0m0.002s
sys 0m0.000s
ygalblum commented
In the Quadlet file you are setting: Image=docker.io/postgres:12-alpine. However, the image FQDN is docker.io/library/postgres:12-alpine (this is what you pull).
podman run knows to translate the incorrect name into the correct one when looking for it locally. But, when it is not available, it tried to download according to the provided value which fails at the server side.
chsasank commented
Tried with that too. Won't work
$ cat /home/sasank/.config/containers/systemd/matrix-postgres.container
[Container]
Image=docker.io/library/postgres:12-alpine
Pod=matrix.pod
EnvironmentFile=/home/sasank/.johnny/matrix/postgres.env
PodmanArgs=
Exec=
Volume=/home/sasank/.johnny/matrix/postgres-data:/var/lib/postgresql/data
[Service]
Restart=always
[Install]
WantedBy=default.target
$ systemctl --user start matrix-postgres.service
Job for matrix-postgres.service canceled.
$ systemctl --user status matrix-postgres.service
○ matrix-postgres.service
Loaded: loaded (/home/sasank/.config/containers/systemd/matrix-postgres.container; generated)
Active: inactive (dead) since Wed 2024-10-09 18:24:16 IST; 4s ago
Process: 3646387 ExecStart=/usr/bin/podman run --name=systemd-matrix-postgres --cidfile=/run/user/1002/matrix-postgres.cid --replace --rm --cgroups=split --sdnotify=conmon -d -v /home/sasank/.johnny/matr>
Process: 3646458 ExecStopPost=/usr/bin/podman rm -v -f -i --cidfile=/run/user/1002/matrix-postgres.cid (code=exited, status=0/SUCCESS)
Main PID: 3646387 (code=exited, status=0/SUCCESS)
CPU: 57ms
Oct 09 18:24:16 JOHNAIC systemd[904]: Starting matrix-postgres.service...
Oct 09 18:24:16 JOHNAIC matrix-postgres[3646387]: Trying to pull docker.io/library/postgres:12-alpine...
Oct 09 18:24:16 JOHNAIC matrix-postgres[3646387]: Pulling image //postgres:12-alpine inside systemd: setting pull timeout to 5m0s
Oct 09 18:24:16 JOHNAIC systemd[904]: Stopped matrix-postgres.service.
ygalblum commented
I tested the Quadlet file and it works on my setup (Fedora 40). So, I think there is something in your setup that's causing it. BTW from the timestamps, I don't think it's a timeout issue because it seems that the service is stopped immediately.
Can you please share your setup? I can see that it depends on other Quadlets.
chsasank commented
How do I share the setup? I have pasted podman info above. I installed podman as in here: https://outline.von-neumann.ai/s/e188eaf2-3269-46d6-a5db-cf95ee9700e2. I use kde neon which is ubuntu 22.04 based release
ygalblum commented
I can see that it depends on other Quadlets
Sorry, if I wasn't clear. I can see that this .container file is only one part of a bigger deployment (it's part of a pod). So, I wanted to get all Quadlet files and their dependencies (e.g. env files) to see if it reproduces on my setup