Podman machine init fails for RHEL 9 with --user-mode-networking
Closed this issue · 12 comments
Bug description
Following using RHEL WSL as a podman machine link I tried to setup a RHEL 9 machine.
Downloaded the tar.gz after installing all required packages from redhat.
Post download, the rhel9 machine does not spin up if user mode networking has been opted for.
Podman was installed using podman-desktop-airgap-1.12.0-setup-x64.exe for a restricted environment sitting behing a VPN and uses proxies. A user mode networking for fedora spins up just fine everytime, but it does not work for RHEL unless podman-net-usermode machine is pre-existing in WSL.
Operating system
Windows 10 Enterprise
Installation Method
Installer from website/GitHub releases
Version
1.12.0
Steps to reproduce
podman desktop version 1.12.0
podman version 5.2.0
$ podman machine init podman-test --user-mode-networking --cpus 4 --memory 1024 --disk-size 100 --rootful --image ./podman-desktop-rhel9-wsl2-v0.tar.gz Extracting compressed file: podman-test-amd64: done Importing operating system into WSL (this may take a few minutes on a new WSL install)... Import in progress, this may take a few minutes. The operation completed successfully. time="2024-09-28T15:17:27+12:00" level=error msg="command C:\\Users\\m810887\\AppData\\Local\\Microsoft\\WindowsApps\\wsl.exe [C:\\Users\\m810887\\AppData\\Local\\Microsoft\\WindowsApps\\wsl.exe --terminate podman-test] failed: exit status 0xffffffff (T\x00h\x00e\x00r\x00e\x00 \x00i\x00s\x00 \x00n\x00o\x00 \x00d\x00i\x00s\x00t\x00r\x00i\x00b\x00u\x00t\x00i\x00o\x00n\x00 \x00w\x00i\x00t\x00h\x00 \x00t\x00h\x00e\x00 \x00s\x00u\x00p\x00p\x00l\x00i\x00e\x00d\x00 \x00n\x00a\x00m\x00e\x00.\x00\r\x00\n\x00E\x00r\x00r\x00o\x00r\x00 \x00c\x00o\x00d\x00e\x00:\x00 \x00W\x00s\x00l\x00/\x00S\x00e\x00r\x00v\x00i\x00c\x00e\x00/\x00W\x00S\x00L\x00_\x00E\x00_\x00D\x00I\x00S\x00T\x00R\x00O\x00_\x00N\x00O\x00T\x00_\x00F\x00O\x00U\x00N\x00D\x00\r\x00\n\x00)" time="2024-09-28T15:17:27+12:00" level=error msg="command C:\\Users\\m810887\\AppData\\Local\\Microsoft\\WindowsApps\\wsl.exe [C:\\Users\\m810887\\AppData\\Local\\Microsoft\\WindowsApps\\wsl.exe --unregister podman-test] failed: exit status 0xffffffff (T\x00h\x00e\x00r\x00e\x00 \x00i\x00s\x00 \x00n\x00o\x00 \x00d\x00i\x00s\x00t\x00r\x00i\x00b\x00u\x00t\x00i\x00o\x00n\x00 \x00w\x00i\x00t\x00h\x00 \x00t\x00h\x00e\x00 \x00s\x00u\x00p\x00p\x00l\x00i\x00e\x00d\x00 \x00n\x00a\x00m\x00e\x00.\x00\r\x00\n\x00E\x00r\x00r\x00o\x00r\x00 \x00c\x00o\x00d\x00e\x00:\x00 \x00W\x00s\x00l\x00/\x00S\x00e\x00r\x00v\x00i\x00c\x00e\x00/\x00W\x00S\x00L\x00_\x00E\x00_\x00D\x00I\x00S\x00T\x00R\x00O\x00_\x00N\x00O\x00T\x00_\x00F\x00O\x00U\x00N\x00D\x00\r\x00\n\x00)" Error: existing machine is too old, can't install user-mode networking dist until machine is reinstalled (using podman machine rm, then podman machine init)
If I remove rootful flag:
$ podman machine init podman-test --user-mode-networking --cpus 4 --memory 1024 --disk-size 100 --image ./podman-desktop-rhel9-wsl2-v0.tar.gz Extracting compressed file: podman-test-amd64: done Importing operating system into WSL (this may take a few minutes on a new WSL install)... Import in progress, this may take a few minutes. The operation completed successfully. time="2024-09-28T15:21:36+12:00" level=error msg="command C:\\Users\\m810887\\AppData\\Local\\Microsoft\\WindowsApps\\wsl.exe [C:\\Users\\m810887\\AppData\\Local\\Microsoft\\WindowsApps\\wsl.exe --terminate podman-test] failed: exit status 0xffffffff (T\x00h\x00e\x00r\x00e\x00 \x00i\x00s\x00 \x00n\x00o\x00 \x00d\x00i\x00s\x00t\x00r\x00i\x00b\x00u\x00t\x00i\x00o\x00n\x00 \x00w\x00i\x00t\x00h\x00 \x00t\x00h\x00e\x00 \x00s\x00u\x00p\x00p\x00l\x00i\x00e\x00d\x00 \x00n\x00a\x00m\x00e\x00.\x00\r\x00\n\x00E\x00r\x00r\x00o\x00r\x00 \x00c\x00o\x00d\x00e\x00:\x00 \x00W\x00s\x00l\x00/\x00S\x00e\x00r\x00v\x00i\x00c\x00e\x00/\x00W\x00S\x00L\x00_\x00E\x00_\x00D\x00I\x00S\x00T\x00R\x00O\x00_\x00N\x00O\x00T\x00_\x00F\x00O\x00U\x00N\x00D\x00\r\x00\n\x00)" time="2024-09-28T15:21:36+12:00" level=error msg="command C:\\Users\\m810887\\AppData\\Local\\Microsoft\\WindowsApps\\wsl.exe [C:\\Users\\m810887\\AppData\\Local\\Microsoft\\WindowsApps\\wsl.exe --unregister podman-test] failed: exit status 0xffffffff (T\x00h\x00e\x00r\x00e\x00 \x00i\x00s\x00 \x00n\x00o\x00 \x00d\x00i\x00s\x00t\x00r\x00i\x00b\x00u\x00t\x00i\x00o\x00n\x00 \x00w\x00i\x00t\x00h\x00 \x00t\x00h\x00e\x00 \x00s\x00u\x00p\x00p\x00l\x00i\x00e\x00d\x00 \x00n\x00a\x00m\x00e\x00.\x00\r\x00\n\x00E\x00r\x00r\x00o\x00r\x00 \x00c\x00o\x00d\x00e\x00:\x00 \x00W\x00s\x00l\x00/\x00S\x00e\x00r\x00v\x00i\x00c\x00e\x00/\x00W\x00S\x00L\x00_\x00E\x00_\x00D\x00I\x00S\x00T\x00R\x00O\x00_\x00N\x00O\x00T\x00_\x00F\x00O\x00U\x00N\x00D\x00\r\x00\n\x00)" Error: existing machine is too old, can't install user-mode networking dist until machine is reinstalled (using podman machine rm, then podman machine init)
But if I remove the --user-mode-networking
flag it installs the machine.
`$ podman machine init podman-test --cpus 4 --memory 1024 --disk-size 100 --rootful --image ./podman-desktop-rhel9-wsl2-v0.tar.gz
Extracting compressed file: podman-test-amd64: done
Importing operating system into WSL (this may take a few minutes on a new WSL install)...
Import in progress, this may take a few minutes.
The operation completed successfully.
Configuring system...
Machine init complete
To start your machine run:
podman machine start podman-test`
Relevant log output
$ podman info
host:
arch: amd64
buildahVersion: 1.33.8
cgroupControllers: []
cgroupManager: cgroupfs
cgroupVersion: v1
conmon:
package: conmon-2.1.10-1.el9.x86_64
path: /usr/bin/conmon
version: 'conmon version 2.1.10, commit: fb8c4bf50dbc044a338137871b096eea8041a1fa'
cpuUtilization:
idlePercent: 99.2
systemPercent: 0.54
userPercent: 0.26
cpus: 8
databaseBackend: sqlite
distribution:
distribution: rhel
version: "9.4"
eventLogger: journald
freeLocks: 2048
hostname: NZ8797LP4535
idMappings:
gidmap:
- container_id: 0
host_id: 1000
size: 1
- container_id: 1
host_id: 100000
size: 65536
uidmap:
- container_id: 0
host_id: 1000
size: 1
- container_id: 1
host_id: 100000
size: 65536
kernel: 5.15.153.1-microsoft-standard-WSL2
linkmode: dynamic
logDriver: journald
memFree: 16023871488
memTotal: 16645128192
networkBackend: netavark
networkBackendInfo:
backend: netavark
dns:
package: Unknown
package: netavark-1.10.3-1.el9.x86_64
path: /usr/libexec/podman/netavark
version: netavark 1.10.3
ociRuntime:
name: crun
package: crun-1.14.3-1.el9.x86_64
path: /usr/bin/crun
version: |-
crun version 1.14.3
commit: 1961d211ba98f532ea52d2e80f4c20359f241a98
rundir: /run/user/1000/crun
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
os: linux
pasta:
executable: ""
package: ""
version: ""
remoteSocket:
exists: true
path: /run/user/1000/podman/podman.sock
rootlessNetworkCmd: ""
security:
apparmorEnabled: false
capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
rootless: true
seccompEnabled: true
seccompProfilePath: /usr/share/containers/seccomp.json
selinuxEnabled: false
serviceIsRemote: true
slirp4netns:
executable: /usr/bin/slirp4netns
package: slirp4netns-1.2.3-1.el9.x86_64
version: |-
slirp4netns version 1.2.3
commit: c22fde291bb35b354e6ca44d13be181c76a0a432
libslirp: 4.4.0
SLIRP_CONFIG_VERSION_MAX: 3
libseccomp: 2.5.2
swapFree: 4294967296
swapTotal: 4294967296
uptime: 0h 22m 56.00s
variant: ""
plugins:
authorization: null
log:
- k8s-file
- none
- passthrough
- journald
network:
- bridge
- macvlan
- ipvlan
volume:
- local
registries:
search:
- docker.io
store:
configFile: /home/user/.config/containers/storage.conf
containerStore:
number: 0
paused: 0
running: 0
stopped: 0
graphDriverName: overlay
graphOptions: {}
graphRoot: /home/user/.local/share/containers/storage
graphRootAllocated: 1081101176832
graphRootUsed: 560078848
graphStatus:
Backing Filesystem: extfs
Native Overlay Diff: "true"
Supports d_type: "true"
Supports shifting: "false"
Supports volatile: "true"
Using metacopy: "false"
imageCopyTmpDir: /var/tmp
imageStore:
number: 0
runRoot: /run/user/1000/containers
transientStore: false
volumePath: /home/user/.local/share/containers/storage/volumes
version:
APIVersion: 4.9.4-rhel
Built: 1723107101
BuiltTime: Thu Aug 8 20:51:41 2024
GitCommit: ""
GoVersion: go1.21.11 (Red Hat 1.21.11-1.el9_4)
Os: linux
OsArch: linux/amd64
Version: 4.9.4-rhel
WSL list:
`$ wsl -l -v
NAME STATE VERSION
* podman-test Stopped 2`
WSL version:
`$ wsl --version
WSL version: 2.2.4.0
Kernel version: 5.15.153.1-2
WSLg version: 1.0.61
MSRDC version: 1.2.5326
Direct3D version: 1.611.1-81528511
DXCore version: 10.0.26091.1-240325-1447.ge-release
Windows version: 10.0.19045.4894`
Additional context
No response
Update:
Something that I noticed was that when the fedora machine spins up using --user-mode-networking
, it spins up another wsl machine for podman-net-usermode
. In case of the rhel machine requiring the --user-mode-networking
, for some reason it is unable to spin that machine up in wsl. So as a test I spun up a fedora machine on podman with --user-mode-networking
, and then unregistered just the fedora machine from wsl and podman. Which then showed that wsl had only 1 machine left in it which was podman-net-usermode
. After that, I tried spinning the RHEL 9 vm up with --rootful and --user-mode-networking
enabled and the machine was up. Just installed the certs and it started working until it was failing again after a reboot.
$ podman --log-level=trace machine init podman-rhel-test --user-mode-networking --rootful --image ./podman-desktop-rhel9-wsl2-v0.tar.gz time="2024-09-29T12:29:27+13:00" level=info msg="C:\\Program Files\\RedHat\\Podman\\podman.exe filtering at log level trace" time="2024-09-29T12:29:27+13:00" level=debug msg="Using Podman machine with
wsl virtualization provider" time="2024-09-29T12:29:27+13:00" level=debug msg="socket length for C:\\Users\\m810887\\.config\\containers\\podman\\machine\\wsl is 54" time="2024-09-29T12:29:27+13:00" level=debug msg="socket length for C:\\Users\\m810887\\.local\\share\\containers\\podman\\machine\\wsl is 59" time="2024-09-29T12:29:27+13:00" level=debug msg="socket length for C:\\Users\\m810887\\.local\\share\\containers\\podman\\machine\\wsl\\cache is 65" time="2024-09-29T12:29:27+13:00" level=debug msg="socket length for C:\\Users\\m810887\\AppData\\Local\\Temp\\podman is 42" time="2024-09-29T12:29:28+13:00" level=debug msg="socket length for C:\\Users\\m810887\\.config\\containers\\podman\\machine\\wsl is 54" time="2024-09-29T12:29:28+13:00" level=debug msg="socket length for C:\\Users\\m810887\\.local\\share\\containers\\podman\\machine\\wsl is 59" time="2024-09-29T12:29:28+13:00" level=debug msg="socket length for C:\\Users\\m810887\\.local\\share\\containers\\podman\\machine\\wsl\\cache is 65" time="2024-09-29T12:29:28+13:00" level=debug msg="socket length for C:\\Users\\m810887\\AppData\\Local\\Temp\\podman is 42" time="2024-09-29T12:29:28+13:00" level=debug msg="socket length for C:\\Users\\m810887\\.config\\containers\\podman\\machine\\wsl\\podman-rhel-test.json is 76" time="2024-09-29T12:29:28+13:00" level=debug msg="socket length for C:\\Users\\m810887\\.local\\share\\containers\\podman\\machine\\wsl\\podman-rhel-test-amd64 is 82" time="2024-09-29T12:29:28+13:00" level=debug msg="socket length for ./podman-desktop-rhel9-wsl2-v0.tar.gz is 37" time="2024-09-29T12:29:28+13:00" level=debug msg="decompressing (if needed) ./podman-desktop-rhel9-wsl2-v0.tar.gz to C:\\Users\\m810887\\.local\\share\\containers\\podman\\machine\\wsl\\podman-rhel-test-amd64" time="2024-09-29T12:29:28+13:00" level=debug msg="Detected compression format gzip" Extracting compressed file: podman-rhel-test-amd64: done time="2024-09-29T12:29:34+13:00" level=debug msg="--> imagePath is \"C:\\\\Users\\\\m810887\\\\.local\\\\share\\\\containers\\\\podman\\\\machine\\\\wsl\\\\podman-rhel-test-amd64\"" time="2024-09-29T12:29:34+13:00" level=debug msg="socket length for C:\\Users\\m810887\\.config\\containers\\podman\\machine\\wsl\\podman-rhel-test.ign is 75" Importing operating system into WSL (this may take a few minutes on a new WSL install)... time="2024-09-29T12:29:34+13:00" level=debug msg="Running command: C:\\Users\\m810887\\AppData\\Local\\Microsoft\\WindowsApps\\wsl.exe [--import podman-rhel-test C:\\Users\\m810887\\.local\\share\\containers\\podman\\machine\\wsl\\wsldist\\podman-rhel-test C:\\Users\\m810887\\.local\\share\\containers\\podman\\machine\\wsl\\podman-rhel-test-amd64 --version 2]" Import in progress, this may take a few minutes. The operation completed successfully. time="2024-09-29T12:29:41+13:00" level=debug msg="Running command: C:\\Users\\m810887\\AppData\\Local\\Microsoft\\WindowsApps\\wsl.exe [-u root -d podman-rhel-test rpm --restore shadow-utils]" time="2024-09-29T12:29:57+13:00" level=debug msg="Running command: C:\\Users\\m810887\\AppData\\Local\\Microsoft\\WindowsApps\\wsl.exe [-u root -d podman-rhel-test mkdir -p /usr/local/bin]" time="2024-09-29T12:29:58+13:00" level=debug msg="Running command: C:\\Users\\m810887\\AppData\\Local\\Microsoft\\WindowsApps\\wsl.exe [-u root -d podman-rhel-test ln -f -s /usr/libexec/podman/gvforwarder /usr/local/bin/vm]" time="2024-09-29T12:29:58+13:00" level=debug msg="Running command: C:\\Users\\m810887\\AppData\\Local\\Microsoft\\WindowsApps\\wsl.exe [-u root -d podman-rhel-test test -f /usr/libexec/podman/gvforwarder]" time="2024-09-29T12:29:59+13:00" level=error msg="command C:\\Users\\m810887\\AppData\\Local\\Microsoft\\WindowsApps\\wsl.exe [C:\\Users\\m810887\\AppData\\Local\\Microsoft\\WindowsApps\\wsl.exe --terminate podman-rhel-test] failed: exit status 0xffffffff (T\x00h\x00e\x00r\x00e\x00 \x00i\x00s\x00 \x00n\x00o\x00 \x00d\x00i\x00s\x00t\x00r\x00i\x00b\x00u\x00t\x00i\x00o\x00n\x00 \x00w\x00i\x00t\x00h\x00 \x00t\x00h\x00e\x00 \x00s\x00u\x00p\x00p\x00l\x00i\x00e\x00d\x00 \x00n\x00a\x00m\x00e\x00.\x00\r\x00\n\x00E\x00r\x00r\x00o\x00r\x00 \x00c\x00o\x00d\x00e\x00:\x00 \x00W\x00s\x00l\x00/\x00S\x00e\x00r\x00v\x00i\x00c\x00e\x00/\x00W\x00S\x00L\x00_\x00E\x00_\x00D\x00I\x00S\x00T\x00R\x00O\x00_\x00N\x00O\x00T\x00_\x00F\x00O\x00U\x00N\x00D\x00\r\x00\n\x00)" time="2024-09-29T12:29:59+13:00" level=error msg="command C:\\Users\\m810887\\AppData\\Local\\Microsoft\\WindowsApps\\wsl.exe [C:\\Users\\m810887\\AppData\\Local\\Microsoft\\WindowsApps\\wsl.exe --unregister podman-rhel-test] failed: exit status 0xffffffff (T\x00h\x00e\x00r\x00e\x00 \x00i\x00s\x00 \x00n\x00o\x00 \x00d\x00i\x00s\x00t\x00r\x00i\x00b\x00u\x00t\x00i\x00o\x00n\x00 \x00w\x00i\x00t\x00h\x00 \x00t\x00h\x00e\x00 \x00s\x00u\x00p\x00p\x00l\x00i\x00e\x00d\x00 \x00n\x00a\x00m\x00e\x00.\x00\r\x00\n\x00E\x00r\x00r\x00o\x00r\x00 \x00c\x00o\x00d\x00e\x00:\x00 \x00W\x00s\x00l\x00/\x00S\x00e\x00r\x00v\x00i\x00c\x00e\x00/\x00W\x00S\x00L\x00_\x00E\x00_\x00D\x00I\x00S\x00T\x00R\x00O\x00_\x00N\x00O\x00T\x00_\x00F\x00O\x00U\x00N\x00D\x00\r\x00\n\x00)" Error: existing machine is too old, can't install user-mode networking dist until machine is reinstalled (using podman machine rm, then podman machine init) time="2024-09-29T12:29:59+13:00" level=debug msg="Shutting down engines"
Ultimately ended up writing a script to create a fedora machine with user mode networking, spin up a rhel 9 machine, and then just delete the fedora machine that leaves the podman net usermode as is.
`
echo "============ START : Fedora"
podman --log-level=debug machine init podman-fedora --user-mode-networking --cpus 4 --memory 1024 --disk-size 100 --rootful --image ./podman-fedora.tar
podman --log-level=debug machine start podman-fedora
echo "=========== FINISH : Fedora"
echo "============ START : RHEL 9"
podman --log-level=debug machine init podman-rhel --user-mode-networking --cpus 4 --memory 1024 --disk-size 100 --rootful --image ./podman-rhel9.tar.gz
podman --log-level=debug machine start podman-rhel
echo "=========== FINISH : RHEL 9"
echo "=========== START : Setting up CERTS"
podman --log-level=debug machine ssh podman-fedora "sudo cp /mnt/c/certs/ca.crt /etc/pki/ca-trust/source/anchors/ && sudo update-ca-trust"
podman --log-level=debug machine ssh podman-rhel "sudo cp /mnt/c/certs/ca.crt /etc/pki/ca-trust/source/anchors/ && sudo update-ca-trust"
echo "========== FINISH : Setting up CERTS"
echo "=========== START : Restarting Machines"
podman --log-level=debug machine stop podman-fedora
podman --log-level=debug machine stop podman-rhel
podman --log-level=debug machine start podman-fedora
podman --log-level=debug machine start podman-rhel
echo "========== FINISH : Restarting Machines"
echo "=========== START : Remove Fedora WSL"
podman --log-level=debug machine stop podman-fedora
podman --log-level=debug machine rm podman-fedora
wsl --unregister podman-fedora
echo "========== FINISH : Remove Fedora WSL"
echo "=========== START : Set Default Machine RHEL"
podman --log-level=debug machine set --default podman-rhel
echo "========== FINISH : Set Default Machine RHEL"
`
After running this, I was able to run podman pull within the RHEL 9 machine.
Hello @nzineer , thank you for creating the issue. I followed the guide you mentioned, and everything worked as expected for me. The Podman machine started successfully in --user-mode-networking
and was running. I was able to pull images without any issues. Therefore, I assume the problem you encountered was related to your environment settings or the RHEL image.
I'm glad to hear that you resolved the issue. Please let me know if I can assist you further. Otherwise, I would prefer to close this issue.
Hey @amisskii , the issue is not resolved for me. The RHEL machine does not spin up if podman-net-usermode doesnt pre-exist. Even if it exists, and the RHEL machine works, it stops working after a reboot of the laptop. Which then leaves me into an undesirable situation of having to rebuild the machine everytime the laptop restarts. So there's nothing that leads me into the direction of a solution at the moment, and not knowing what the errors indicate complicates this even further.
Can reproduce on my Win11Pro laptop with Podman 5.2.2
I’ve managed to reproduce the issue too.
Moving to upstream podman to make the team aware even if it's a non supported configuration
The podman team only supports the default fedora based wsl distro.
The error that is mentioned checks for a specific binary in the image /usr/libexec/podman/gvforwarder
so I guess you did not have the right dependencies in there, in any case this is not something we support so I close this one
@jeffmaury since this was closed off here, would this need to go back to podman-desktop then?
I would not open an issue as it's not related to podm desktop per se but we can have a discussion on the podman desktop repo
fwiw, the package in fedora is called gvisor-tap-vsock-gvforwarder.x86_64
... I havent looked but assume it is not being built as part of RHEL?
actually it is in gvisor-tap-vsock
/usr/lib/.build-id
/usr/lib/.build-id/a5
/usr/lib/.build-id/a5/15be63df7aff0b03e5c8a115b34e13f97bde14
/usr/lib/.build-id/eb
/usr/lib/.build-id/eb/10d2c8b70ba7ecc17daf70cd8ac231135546d8
/usr/libexec/podman
/usr/libexec/podman/gvforwarder
/usr/libexec/podman/gvproxy
/usr/share/doc/gvisor-tap-vsock
/usr/share/doc/gvisor-tap-vsock/README.md
/usr/share/licenses/gvisor-tap-vsock
/usr/share/licenses/gvisor-tap-vsock/LICENSE
Though as @Luap99 points out, this is well out of what we are willing to support.