containers/podman

Podman machine init fails for RHEL 9 with --user-mode-networking

Closed this issue · 12 comments

Bug description

Following using RHEL WSL as a podman machine link I tried to setup a RHEL 9 machine.
Downloaded the tar.gz after installing all required packages from redhat.
Post download, the rhel9 machine does not spin up if user mode networking has been opted for.
Podman was installed using podman-desktop-airgap-1.12.0-setup-x64.exe for a restricted environment sitting behing a VPN and uses proxies. A user mode networking for fedora spins up just fine everytime, but it does not work for RHEL unless podman-net-usermode machine is pre-existing in WSL.

Operating system

Windows 10 Enterprise

Installation Method

Installer from website/GitHub releases

Version

1.12.0

Steps to reproduce

podman desktop version 1.12.0
podman version 5.2.0

$ podman machine init podman-test --user-mode-networking --cpus 4 --memory 1024 --disk-size 100 --rootful --image ./podman-desktop-rhel9-wsl2-v0.tar.gz Extracting compressed file: podman-test-amd64: done Importing operating system into WSL (this may take a few minutes on a new WSL install)... Import in progress, this may take a few minutes. The operation completed successfully. time="2024-09-28T15:17:27+12:00" level=error msg="command C:\\Users\\m810887\\AppData\\Local\\Microsoft\\WindowsApps\\wsl.exe [C:\\Users\\m810887\\AppData\\Local\\Microsoft\\WindowsApps\\wsl.exe --terminate podman-test] failed: exit status 0xffffffff (T\x00h\x00e\x00r\x00e\x00 \x00i\x00s\x00 \x00n\x00o\x00 \x00d\x00i\x00s\x00t\x00r\x00i\x00b\x00u\x00t\x00i\x00o\x00n\x00 \x00w\x00i\x00t\x00h\x00 \x00t\x00h\x00e\x00 \x00s\x00u\x00p\x00p\x00l\x00i\x00e\x00d\x00 \x00n\x00a\x00m\x00e\x00.\x00\r\x00\n\x00E\x00r\x00r\x00o\x00r\x00 \x00c\x00o\x00d\x00e\x00:\x00 \x00W\x00s\x00l\x00/\x00S\x00e\x00r\x00v\x00i\x00c\x00e\x00/\x00W\x00S\x00L\x00_\x00E\x00_\x00D\x00I\x00S\x00T\x00R\x00O\x00_\x00N\x00O\x00T\x00_\x00F\x00O\x00U\x00N\x00D\x00\r\x00\n\x00)" time="2024-09-28T15:17:27+12:00" level=error msg="command C:\\Users\\m810887\\AppData\\Local\\Microsoft\\WindowsApps\\wsl.exe [C:\\Users\\m810887\\AppData\\Local\\Microsoft\\WindowsApps\\wsl.exe --unregister podman-test] failed: exit status 0xffffffff (T\x00h\x00e\x00r\x00e\x00 \x00i\x00s\x00 \x00n\x00o\x00 \x00d\x00i\x00s\x00t\x00r\x00i\x00b\x00u\x00t\x00i\x00o\x00n\x00 \x00w\x00i\x00t\x00h\x00 \x00t\x00h\x00e\x00 \x00s\x00u\x00p\x00p\x00l\x00i\x00e\x00d\x00 \x00n\x00a\x00m\x00e\x00.\x00\r\x00\n\x00E\x00r\x00r\x00o\x00r\x00 \x00c\x00o\x00d\x00e\x00:\x00 \x00W\x00s\x00l\x00/\x00S\x00e\x00r\x00v\x00i\x00c\x00e\x00/\x00W\x00S\x00L\x00_\x00E\x00_\x00D\x00I\x00S\x00T\x00R\x00O\x00_\x00N\x00O\x00T\x00_\x00F\x00O\x00U\x00N\x00D\x00\r\x00\n\x00)" Error: existing machine is too old, can't install user-mode networking dist until machine is reinstalled (using podman machine rm, then podman machine init)

If I remove rootful flag:
$ podman machine init podman-test --user-mode-networking --cpus 4 --memory 1024 --disk-size 100 --image ./podman-desktop-rhel9-wsl2-v0.tar.gz Extracting compressed file: podman-test-amd64: done Importing operating system into WSL (this may take a few minutes on a new WSL install)... Import in progress, this may take a few minutes. The operation completed successfully. time="2024-09-28T15:21:36+12:00" level=error msg="command C:\\Users\\m810887\\AppData\\Local\\Microsoft\\WindowsApps\\wsl.exe [C:\\Users\\m810887\\AppData\\Local\\Microsoft\\WindowsApps\\wsl.exe --terminate podman-test] failed: exit status 0xffffffff (T\x00h\x00e\x00r\x00e\x00 \x00i\x00s\x00 \x00n\x00o\x00 \x00d\x00i\x00s\x00t\x00r\x00i\x00b\x00u\x00t\x00i\x00o\x00n\x00 \x00w\x00i\x00t\x00h\x00 \x00t\x00h\x00e\x00 \x00s\x00u\x00p\x00p\x00l\x00i\x00e\x00d\x00 \x00n\x00a\x00m\x00e\x00.\x00\r\x00\n\x00E\x00r\x00r\x00o\x00r\x00 \x00c\x00o\x00d\x00e\x00:\x00 \x00W\x00s\x00l\x00/\x00S\x00e\x00r\x00v\x00i\x00c\x00e\x00/\x00W\x00S\x00L\x00_\x00E\x00_\x00D\x00I\x00S\x00T\x00R\x00O\x00_\x00N\x00O\x00T\x00_\x00F\x00O\x00U\x00N\x00D\x00\r\x00\n\x00)" time="2024-09-28T15:21:36+12:00" level=error msg="command C:\\Users\\m810887\\AppData\\Local\\Microsoft\\WindowsApps\\wsl.exe [C:\\Users\\m810887\\AppData\\Local\\Microsoft\\WindowsApps\\wsl.exe --unregister podman-test] failed: exit status 0xffffffff (T\x00h\x00e\x00r\x00e\x00 \x00i\x00s\x00 \x00n\x00o\x00 \x00d\x00i\x00s\x00t\x00r\x00i\x00b\x00u\x00t\x00i\x00o\x00n\x00 \x00w\x00i\x00t\x00h\x00 \x00t\x00h\x00e\x00 \x00s\x00u\x00p\x00p\x00l\x00i\x00e\x00d\x00 \x00n\x00a\x00m\x00e\x00.\x00\r\x00\n\x00E\x00r\x00r\x00o\x00r\x00 \x00c\x00o\x00d\x00e\x00:\x00 \x00W\x00s\x00l\x00/\x00S\x00e\x00r\x00v\x00i\x00c\x00e\x00/\x00W\x00S\x00L\x00_\x00E\x00_\x00D\x00I\x00S\x00T\x00R\x00O\x00_\x00N\x00O\x00T\x00_\x00F\x00O\x00U\x00N\x00D\x00\r\x00\n\x00)" Error: existing machine is too old, can't install user-mode networking dist until machine is reinstalled (using podman machine rm, then podman machine init)

But if I remove the --user-mode-networking flag it installs the machine.
`$ podman machine init podman-test --cpus 4 --memory 1024 --disk-size 100 --rootful --image ./podman-desktop-rhel9-wsl2-v0.tar.gz
Extracting compressed file: podman-test-amd64: done
Importing operating system into WSL (this may take a few minutes on a new WSL install)...
Import in progress, this may take a few minutes.
The operation completed successfully.
Configuring system...
Machine init complete
To start your machine run:

    podman machine start podman-test`

Relevant log output

$ podman info
host:
  arch: amd64
  buildahVersion: 1.33.8
  cgroupControllers: []
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: conmon-2.1.10-1.el9.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.10, commit: fb8c4bf50dbc044a338137871b096eea8041a1fa'
  cpuUtilization:
    idlePercent: 99.2
    systemPercent: 0.54
    userPercent: 0.26
  cpus: 8
  databaseBackend: sqlite
  distribution:
    distribution: rhel
    version: "9.4"
  eventLogger: journald
  freeLocks: 2048
  hostname: NZ8797LP4535
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.15.153.1-microsoft-standard-WSL2
  linkmode: dynamic
  logDriver: journald
  memFree: 16023871488
  memTotal: 16645128192
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: Unknown
    package: netavark-1.10.3-1.el9.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.10.3
  ociRuntime:
    name: crun
    package: crun-1.14.3-1.el9.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.14.3
      commit: 1961d211ba98f532ea52d2e80f4c20359f241a98
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  pasta:
    executable: ""
    package: ""
    version: ""
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  rootlessNetworkCmd: ""
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.3-1.el9.x86_64
    version: |-
      slirp4netns version 1.2.3
      commit: c22fde291bb35b354e6ca44d13be181c76a0a432
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.2
  swapFree: 4294967296
  swapTotal: 4294967296
  uptime: 0h 22m 56.00s
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /home/user/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/user/.local/share/containers/storage
  graphRootAllocated: 1081101176832
  graphRootUsed: 560078848
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 0
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /home/user/.local/share/containers/storage/volumes
version:
  APIVersion: 4.9.4-rhel
  Built: 1723107101
  BuiltTime: Thu Aug  8 20:51:41 2024
  GitCommit: ""
  GoVersion: go1.21.11 (Red Hat 1.21.11-1.el9_4)
  Os: linux
  OsArch: linux/amd64
  Version: 4.9.4-rhel


WSL list:
`$ wsl -l -v
  NAME           STATE           VERSION
* podman-test    Stopped         2`

WSL version:
`$ wsl --version
WSL version: 2.2.4.0
Kernel version: 5.15.153.1-2
WSLg version: 1.0.61
MSRDC version: 1.2.5326
Direct3D version: 1.611.1-81528511
DXCore version: 10.0.26091.1-240325-1447.ge-release
Windows version: 10.0.19045.4894`

Additional context

No response

Update:

Something that I noticed was that when the fedora machine spins up using --user-mode-networking, it spins up another wsl machine for podman-net-usermode. In case of the rhel machine requiring the --user-mode-networking, for some reason it is unable to spin that machine up in wsl. So as a test I spun up a fedora machine on podman with --user-mode-networking, and then unregistered just the fedora machine from wsl and podman. Which then showed that wsl had only 1 machine left in it which was podman-net-usermode. After that, I tried spinning the RHEL 9 vm up with --rootful and --user-mode-networking enabled and the machine was up. Just installed the certs and it started working until it was failing again after a reboot.

$ podman --log-level=trace machine init podman-rhel-test --user-mode-networking --rootful --image ./podman-desktop-rhel9-wsl2-v0.tar.gz time="2024-09-29T12:29:27+13:00" level=info msg="C:\\Program Files\\RedHat\\Podman\\podman.exe filtering at log level trace" time="2024-09-29T12:29:27+13:00" level=debug msg="Using Podman machine with wsl virtualization provider" time="2024-09-29T12:29:27+13:00" level=debug msg="socket length for C:\\Users\\m810887\\.config\\containers\\podman\\machine\\wsl is 54" time="2024-09-29T12:29:27+13:00" level=debug msg="socket length for C:\\Users\\m810887\\.local\\share\\containers\\podman\\machine\\wsl is 59" time="2024-09-29T12:29:27+13:00" level=debug msg="socket length for C:\\Users\\m810887\\.local\\share\\containers\\podman\\machine\\wsl\\cache is 65" time="2024-09-29T12:29:27+13:00" level=debug msg="socket length for C:\\Users\\m810887\\AppData\\Local\\Temp\\podman is 42" time="2024-09-29T12:29:28+13:00" level=debug msg="socket length for C:\\Users\\m810887\\.config\\containers\\podman\\machine\\wsl is 54" time="2024-09-29T12:29:28+13:00" level=debug msg="socket length for C:\\Users\\m810887\\.local\\share\\containers\\podman\\machine\\wsl is 59" time="2024-09-29T12:29:28+13:00" level=debug msg="socket length for C:\\Users\\m810887\\.local\\share\\containers\\podman\\machine\\wsl\\cache is 65" time="2024-09-29T12:29:28+13:00" level=debug msg="socket length for C:\\Users\\m810887\\AppData\\Local\\Temp\\podman is 42" time="2024-09-29T12:29:28+13:00" level=debug msg="socket length for C:\\Users\\m810887\\.config\\containers\\podman\\machine\\wsl\\podman-rhel-test.json is 76" time="2024-09-29T12:29:28+13:00" level=debug msg="socket length for C:\\Users\\m810887\\.local\\share\\containers\\podman\\machine\\wsl\\podman-rhel-test-amd64 is 82" time="2024-09-29T12:29:28+13:00" level=debug msg="socket length for ./podman-desktop-rhel9-wsl2-v0.tar.gz is 37" time="2024-09-29T12:29:28+13:00" level=debug msg="decompressing (if needed) ./podman-desktop-rhel9-wsl2-v0.tar.gz to C:\\Users\\m810887\\.local\\share\\containers\\podman\\machine\\wsl\\podman-rhel-test-amd64" time="2024-09-29T12:29:28+13:00" level=debug msg="Detected compression format gzip" Extracting compressed file: podman-rhel-test-amd64: done time="2024-09-29T12:29:34+13:00" level=debug msg="--> imagePath is \"C:\\\\Users\\\\m810887\\\\.local\\\\share\\\\containers\\\\podman\\\\machine\\\\wsl\\\\podman-rhel-test-amd64\"" time="2024-09-29T12:29:34+13:00" level=debug msg="socket length for C:\\Users\\m810887\\.config\\containers\\podman\\machine\\wsl\\podman-rhel-test.ign is 75" Importing operating system into WSL (this may take a few minutes on a new WSL install)... time="2024-09-29T12:29:34+13:00" level=debug msg="Running command: C:\\Users\\m810887\\AppData\\Local\\Microsoft\\WindowsApps\\wsl.exe [--import podman-rhel-test C:\\Users\\m810887\\.local\\share\\containers\\podman\\machine\\wsl\\wsldist\\podman-rhel-test C:\\Users\\m810887\\.local\\share\\containers\\podman\\machine\\wsl\\podman-rhel-test-amd64 --version 2]" Import in progress, this may take a few minutes. The operation completed successfully. time="2024-09-29T12:29:41+13:00" level=debug msg="Running command: C:\\Users\\m810887\\AppData\\Local\\Microsoft\\WindowsApps\\wsl.exe [-u root -d podman-rhel-test rpm --restore shadow-utils]" time="2024-09-29T12:29:57+13:00" level=debug msg="Running command: C:\\Users\\m810887\\AppData\\Local\\Microsoft\\WindowsApps\\wsl.exe [-u root -d podman-rhel-test mkdir -p /usr/local/bin]" time="2024-09-29T12:29:58+13:00" level=debug msg="Running command: C:\\Users\\m810887\\AppData\\Local\\Microsoft\\WindowsApps\\wsl.exe [-u root -d podman-rhel-test ln -f -s /usr/libexec/podman/gvforwarder /usr/local/bin/vm]" time="2024-09-29T12:29:58+13:00" level=debug msg="Running command: C:\\Users\\m810887\\AppData\\Local\\Microsoft\\WindowsApps\\wsl.exe [-u root -d podman-rhel-test test -f /usr/libexec/podman/gvforwarder]" time="2024-09-29T12:29:59+13:00" level=error msg="command C:\\Users\\m810887\\AppData\\Local\\Microsoft\\WindowsApps\\wsl.exe [C:\\Users\\m810887\\AppData\\Local\\Microsoft\\WindowsApps\\wsl.exe --terminate podman-rhel-test] failed: exit status 0xffffffff (T\x00h\x00e\x00r\x00e\x00 \x00i\x00s\x00 \x00n\x00o\x00 \x00d\x00i\x00s\x00t\x00r\x00i\x00b\x00u\x00t\x00i\x00o\x00n\x00 \x00w\x00i\x00t\x00h\x00 \x00t\x00h\x00e\x00 \x00s\x00u\x00p\x00p\x00l\x00i\x00e\x00d\x00 \x00n\x00a\x00m\x00e\x00.\x00\r\x00\n\x00E\x00r\x00r\x00o\x00r\x00 \x00c\x00o\x00d\x00e\x00:\x00 \x00W\x00s\x00l\x00/\x00S\x00e\x00r\x00v\x00i\x00c\x00e\x00/\x00W\x00S\x00L\x00_\x00E\x00_\x00D\x00I\x00S\x00T\x00R\x00O\x00_\x00N\x00O\x00T\x00_\x00F\x00O\x00U\x00N\x00D\x00\r\x00\n\x00)" time="2024-09-29T12:29:59+13:00" level=error msg="command C:\\Users\\m810887\\AppData\\Local\\Microsoft\\WindowsApps\\wsl.exe [C:\\Users\\m810887\\AppData\\Local\\Microsoft\\WindowsApps\\wsl.exe --unregister podman-rhel-test] failed: exit status 0xffffffff (T\x00h\x00e\x00r\x00e\x00 \x00i\x00s\x00 \x00n\x00o\x00 \x00d\x00i\x00s\x00t\x00r\x00i\x00b\x00u\x00t\x00i\x00o\x00n\x00 \x00w\x00i\x00t\x00h\x00 \x00t\x00h\x00e\x00 \x00s\x00u\x00p\x00p\x00l\x00i\x00e\x00d\x00 \x00n\x00a\x00m\x00e\x00.\x00\r\x00\n\x00E\x00r\x00r\x00o\x00r\x00 \x00c\x00o\x00d\x00e\x00:\x00 \x00W\x00s\x00l\x00/\x00S\x00e\x00r\x00v\x00i\x00c\x00e\x00/\x00W\x00S\x00L\x00_\x00E\x00_\x00D\x00I\x00S\x00T\x00R\x00O\x00_\x00N\x00O\x00T\x00_\x00F\x00O\x00U\x00N\x00D\x00\r\x00\n\x00)" Error: existing machine is too old, can't install user-mode networking dist until machine is reinstalled (using podman machine rm, then podman machine init) time="2024-09-29T12:29:59+13:00" level=debug msg="Shutting down engines"

Ultimately ended up writing a script to create a fedora machine with user mode networking, spin up a rhel 9 machine, and then just delete the fedora machine that leaves the podman net usermode as is.

`
echo "============ START : Fedora"
podman --log-level=debug machine init podman-fedora --user-mode-networking --cpus 4 --memory 1024 --disk-size 100 --rootful --image ./podman-fedora.tar
podman --log-level=debug machine start podman-fedora
echo "=========== FINISH : Fedora"

echo "============ START : RHEL 9"
podman --log-level=debug machine init podman-rhel --user-mode-networking --cpus 4 --memory 1024 --disk-size 100 --rootful --image ./podman-rhel9.tar.gz
podman --log-level=debug machine start podman-rhel
echo "=========== FINISH : RHEL 9"

echo "=========== START : Setting up CERTS"
podman --log-level=debug machine ssh podman-fedora "sudo cp /mnt/c/certs/ca.crt /etc/pki/ca-trust/source/anchors/ && sudo update-ca-trust"
podman --log-level=debug machine ssh podman-rhel "sudo cp /mnt/c/certs/ca.crt /etc/pki/ca-trust/source/anchors/ && sudo update-ca-trust"
echo "========== FINISH : Setting up CERTS"

echo "=========== START : Restarting Machines"
podman --log-level=debug machine stop podman-fedora
podman --log-level=debug machine stop podman-rhel
podman --log-level=debug machine start podman-fedora
podman --log-level=debug machine start podman-rhel
echo "========== FINISH : Restarting Machines"

echo "=========== START : Remove Fedora WSL"
podman --log-level=debug machine stop podman-fedora
podman --log-level=debug machine rm podman-fedora
wsl --unregister podman-fedora
echo "========== FINISH : Remove Fedora WSL"

echo "=========== START : Set Default Machine RHEL"
podman --log-level=debug machine set --default podman-rhel
echo "========== FINISH : Set Default Machine RHEL"

`

After running this, I was able to run podman pull within the RHEL 9 machine.

Hello @nzineer , thank you for creating the issue. I followed the guide you mentioned, and everything worked as expected for me. The Podman machine started successfully in --user-mode-networking and was running. I was able to pull images without any issues. Therefore, I assume the problem you encountered was related to your environment settings or the RHEL image.

I'm glad to hear that you resolved the issue. Please let me know if I can assist you further. Otherwise, I would prefer to close this issue.

Hey @amisskii , the issue is not resolved for me. The RHEL machine does not spin up if podman-net-usermode doesnt pre-exist. Even if it exists, and the RHEL machine works, it stops working after a reboot of the laptop. Which then leaves me into an undesirable situation of having to rebuild the machine everytime the laptop restarts. So there's nothing that leads me into the direction of a solution at the moment, and not knowing what the errors indicate complicates this even further.

Can reproduce on my Win11Pro laptop with Podman 5.2.2

I’ve managed to reproduce the issue too.

Moving to upstream podman to make the team aware even if it's a non supported configuration

The podman team only supports the default fedora based wsl distro.
The error that is mentioned checks for a specific binary in the image /usr/libexec/podman/gvforwarder so I guess you did not have the right dependencies in there, in any case this is not something we support so I close this one

@jeffmaury since this was closed off here, would this need to go back to podman-desktop then?

I would not open an issue as it's not related to podm desktop per se but we can have a discussion on the podman desktop repo

baude commented

fwiw, the package in fedora is called gvisor-tap-vsock-gvforwarder.x86_64 ... I havent looked but assume it is not being built as part of RHEL?

baude commented

actually it is in gvisor-tap-vsock

/usr/lib/.build-id
/usr/lib/.build-id/a5
/usr/lib/.build-id/a5/15be63df7aff0b03e5c8a115b34e13f97bde14
/usr/lib/.build-id/eb
/usr/lib/.build-id/eb/10d2c8b70ba7ecc17daf70cd8ac231135546d8
/usr/libexec/podman
/usr/libexec/podman/gvforwarder
/usr/libexec/podman/gvproxy
/usr/share/doc/gvisor-tap-vsock
/usr/share/doc/gvisor-tap-vsock/README.md
/usr/share/licenses/gvisor-tap-vsock
/usr/share/licenses/gvisor-tap-vsock/LICENSE

Though as @Luap99 points out, this is well out of what we are willing to support.