[General Issue]: PiVPN blocks gateway from containers
tholeb opened this issue · 4 comments
tholeb commented
In raising this issue I confirm that
- I have read the documentation
- I have read and understood the PiVPN General Guidelines
- I have read and understood the PiVPN Troubleshooting Guidelines
- The issue I am reporting isn't a duplicate, see closed issues and open issues.
- I have searched for similar issues and solutions
- I can replicate the issue even after a clean OS installation
Describe the issue
It seems that PiVPN blocks some gateway inside containers.
Let me explain. I have a container that is a simple website (lighttpd) on <localip>:8080. I have a nginx server (not container) that proxy example.com to <localip>:8080. When there is no issue, I can go both and have a perfect webpage, working as intended.
But, now that pivpn is installed, when I go to this container using the domain, I get a 502 bad gateway, and a "connection refused" when I try to curl it (both, domain and local ip).
I had this problem for a few weeks (month ?), I even opened an issue on podman, so I decided to reinstall my pi today. Everything went well, I had no error, and no problems whatsoever, I even thought that this clean installation had fixed my issue, but I just reinstalled pivpn, and after the system restart (in script), my container is not working anymore (502 bad gateway), so i'm 100% sure that this issue comes from pivpn.
Additional information :
Here is how I deploy the container :
- name: Homer - Pull Docker image
containers.podman.podman_image:
name: docker.io/b4bz/homer
pull: true
force: true
tag: latest
- name: Homer - Run container using podman
containers.podman.podman_container:
name: homer
image: b4bz/homer:latest
state: started
recreate: true
restart_policy: on-failure
user: 0:0
volumes:
- "{{ homer_data }}:/www/assets:rw"
ports: "8080:8080"
memory: "512m"
generate_systemd:
path: /etc/systemd/system
new: true
notify:
- Systemd daemon reload
- Restart homerOutput of podman info
host:
arch: arm64
buildahVersion: 1.23.1
cgroupControllers:
- cpuset
- cpu
- io
- memory
- hugetlb
- pids
- rdma
- misc
cgroupManager: systemd
cgroupVersion: v2
conmon:
package: 'conmon: /usr/bin/conmon'
path: /usr/bin/conmon
version: 'conmon version 2.0.25, commit: unknown'
cpus: 4
distribution:
codename: jammy
distribution: ubuntu
version: "22.04"
eventLogger: journald
hostname: raspberry
idMappings:
gidmap: null
uidmap: null
kernel: 5.15.0-1027-raspi
linkmode: dynamic
logDriver: journald
memFree: 6287241216
memTotal: 8186318848
ociRuntime:
name: crun
package: 'crun: /usr/bin/crun'
path: /usr/bin/crun
version: |-
crun version 0.17
commit: 0e9229ae34caaebcb86f1fde18de3acaf18c6d9a
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
os: linux
remoteSocket:
exists: true
path: /run/podman/podman.sock
security:
apparmorEnabled: true
capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
rootless: false
seccompEnabled: true
seccompProfilePath: /usr/share/containers/seccomp.json
selinuxEnabled: false
serviceIsRemote: false
slirp4netns:
executable: /usr/bin/slirp4netns
package: 'slirp4netns: /usr/bin/slirp4netns'
version: |-
slirp4netns version 1.0.1
commit: 6a7b16babc95b6a3056b33fb45b74a6f62262dd4
libslirp: 4.6.1
swapFree: 0
swapTotal: 0
uptime: 1h 48m 26.02s (Approximately 0.04 days)
plugins:
log:
- k8s-file
- none
- journald
network:
- bridge
- macvlan
volume:
- local
registries: {}
store:
configFile: /etc/containers/storage.conf
containerStore:
number: 8
paused: 0
running: 8
stopped: 0
graphDriverName: overlay
graphOptions: {}
graphRoot: /var/lib/containers/storage
graphStatus:
Backing Filesystem: extfs
Native Overlay Diff: "true"
Supports d_type: "true"
Using metacopy: "false"
imageStore:
number: 8
runRoot: /run/containers/storage
volumePath: /var/lib/containers/storage/volumes
version:
APIVersion: 3.4.4
Built: 0
BuiltTime: Thu Jan 1 01:00:00 1970
GitCommit: ""
GoVersion: go1.17.3
OsArch: linux/arm64
Version: 3.4.4Expected behavior
I expect my container to work properly.
Please describe the steps to replicate the issue
- install and start homer
- check that the container is working properly, restart it and check again
- download and install pivpn (restart the pi as suggested)
- Check the container. It should not load properly ("You're offline friend. ") and inspect the console, and you should get multiple 502.
Have you taken any steps towards solving your issue?
Yes, I tried looking for anything related on google, but nothing relevant. I also tried debugging on the container (ip a, netstat, ...) and nothing seams problematic (thought, I don't really know a lot about networking, so I may be wrong).
I also opened an issue on podman's repo.
Screenshots
Container not working :
Where did you run pivpn?
Raspberry Pi 4 8Gb
Please provide your output from uname -a
Linux ubuntu 5.15.0-1027-raspi #29-Ubuntu SMP PREEMPT Mon Apr 3 10:12:21 UTC 2023 aarch64 aarch64 aarch64 GNU/Linux
Details about Operative System
PRETTY_NAME="Ubuntu 22.04.2 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.2 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy
Installation
Profile / Client creation
Adds a vpn user as intended. No issue there.
Debug output
::: Generating Debug Output
:::: PiVPN debug ::::
=============================================
:::: Latest commit ::::
Branch: master
Commit: 4032a55c80f25b51419180eda93f44d579ab79e9
Author: 4s3ti
Date: Wed Mar 29 14:54:19 2023 +0200
Summary: docs(issues): Remove old markdown template
=============================================
:::: Installation settings ::::
PLAT=Ubuntu
OSCN=jammy
USING_UFW=0
pivpnforceipv6route=0
IPv4dev=eth0
install_user=vpn
install_home=/home/vpn
VPN=wireguard
pivpnPORT=51820
pivpnDNS1=192.168.1.10
pivpnDNS2=192.168.1.1
pivpnHOST=REDACTED
INPUT_CHAIN_EDITED=1
FORWARD_CHAIN_EDITED=1
INPUT_CHAIN_EDITEDv6=
FORWARD_CHAIN_EDITEDv6=
pivpnPROTO=udp
pivpnMTU=1420
pivpnDEV=wg0
pivpnNET=10.134.206.0
subnetClass=24
pivpnenableipv6=0
ALLOWED_IPS="0.0.0.0/0"
UNATTUPG=1
INSTALLED_PACKAGES=(dnsutils grepcidr bsdmainutils iptables-persistent wireguard-tools qrencode)
=============================================
:::: Server configuration shown below ::::
[Interface]
PrivateKey = server_priv
Address = 10.134.206.1/24
MTU = 1420
ListenPort = 51820
=============================================
:::: Client configuration shown below ::::
::: There are no clients yet
=============================================
:::: Recursive list of files in ::::
:::: /etc/wireguard shown below ::::
/etc/wireguard:
configs
keys
wg0.conf
/etc/wireguard/configs:
clients.txt
/etc/wireguard/keys:
server_priv
server_pub
=============================================
:::: Self check ::::
:: [OK] IP forwarding is enabled
:: [OK] Iptables MASQUERADE rule set
:: [OK] Iptables INPUT rule set
:: [OK] Iptables FORWARD rule set
:: [OK] WireGuard is running
:: [OK] WireGuard is enabled
(it will automatically start on reboot)
:: [OK] WireGuard is listening on port 51820/udp
=============================================
:::: Having trouble connecting? Take a look at the FAQ:
:::: https://docs.pivpn.io/faq
=============================================
:::: WARNING: This script should have automatically masked sensitive ::::
:::: information, however, still make sure that PrivateKey, PublicKey ::::
:::: and PresharedKey are masked before reporting an issue. An example key ::::
:::: that you should NOT see in this log looks like this: ::::
:::: YIAoJVsdIeyvXfGGDDadHh6AxsMRymZTnnzZoAb9cxRe ::::
=============================================
:::: Debug complete ::::
:::
::: Debug output completed above.
::: Copy saved to /tmp/debug.log
:::
orazioedoardo commented
Might have to do with the firewall. Can you post the output of iptables -vnL and iptables -vnL -t nat ?
tholeb commented
iptables -vnL
Chain INPUT (policy ACCEPT 11M packets, 1703M bytes)
pkts bytes target prot opt in out source destination
130K 29M ACCEPT udp -- eth0 * 0.0.0.0/0 0.0.0.0/0 udp dpt:51820 /* wireguard-input-rule */
11M 1703M LIBVIRT_INP all -- * * 0.0.0.0/0 0.0.0.0/0
Chain FORWARD (policy ACCEPT 47 packets, 2580 bytes)
pkts bytes target prot opt in out source destination
419K 526M ACCEPT all -- eth0 wg0 0.0.0.0/0 10.134.206.0/24 ctstate RELATED,ESTABLISHED /* wireguard-forward-rule */
103K 11M ACCEPT all -- wg0 eth0 10.134.206.0/24 0.0.0.0/0 /* wireguard-forward-rule */
473 2154K CNI-FORWARD all -- * * 0.0.0.0/0 0.0.0.0/0 /* CNI firewall plugin rules */
47 2580 LIBVIRT_FWX all -- * * 0.0.0.0/0 0.0.0.0/0
47 2580 LIBVIRT_FWI all -- * * 0.0.0.0/0 0.0.0.0/0
47 2580 LIBVIRT_FWO all -- * * 0.0.0.0/0 0.0.0.0/0
Chain OUTPUT (policy ACCEPT 9730K packets, 6498M bytes)
pkts bytes target prot opt in out source destination
9730K 6498M LIBVIRT_OUT all -- * * 0.0.0.0/0 0.0.0.0/0
Chain CNI-ADMIN (1 references)
pkts bytes target prot opt in out source destination
Chain CNI-FORWARD (1 references)
pkts bytes target prot opt in out source destination
473 2154K CNI-ADMIN all -- * * 0.0.0.0/0 0.0.0.0/0 /* CNI firewall plugin admin overrides */
0 0 ACCEPT all -- * * 0.0.0.0/0 10.88.0.3 ctstate RELATED,ESTABLISHED
0 0 ACCEPT all -- * * 10.88.0.3 0.0.0.0/0
0 0 ACCEPT all -- * * 0.0.0.0/0 10.88.0.4 ctstate RELATED,ESTABLISHED
0 0 ACCEPT all -- * * 10.88.0.4 0.0.0.0/0
338 2146K ACCEPT all -- * * 0.0.0.0/0 10.88.0.5 ctstate RELATED,ESTABLISHED
88 5038 ACCEPT all -- * * 10.88.0.5 0.0.0.0/0
0 0 ACCEPT all -- * * 0.0.0.0/0 10.88.0.23 ctstate RELATED,ESTABLISHED
0 0 ACCEPT all -- * * 10.88.0.23 0.0.0.0/0
0 0 ACCEPT all -- * * 0.0.0.0/0 10.88.0.7 ctstate RELATED,ESTABLISHED
0 0 ACCEPT all -- * * 10.88.0.7 0.0.0.0/0
Chain LIBVIRT_FWI (1 references)
pkts bytes target prot opt in out source destination
0 0 ACCEPT all -- * virbr0 0.0.0.0/0 192.168.122.0/24 ctstate RELATED,ESTABLISHED
0 0 REJECT all -- * virbr0 0.0.0.0/0 0.0.0.0/0 reject-with icmp-port-unreachable
0 0 ACCEPT all -- * virbr0 0.0.0.0/0 192.168.122.0/24 ctstate RELATED,ESTABLISHED
0 0 REJECT all -- * virbr0 0.0.0.0/0 0.0.0.0/0 reject-with icmp-port-unreachable
Chain LIBVIRT_FWO (1 references)
pkts bytes target prot opt in out source destination
0 0 ACCEPT all -- virbr0 * 192.168.122.0/24 0.0.0.0/0
0 0 REJECT all -- virbr0 * 0.0.0.0/0 0.0.0.0/0 reject-with icmp-port-unreachable
0 0 ACCEPT all -- virbr0 * 192.168.122.0/24 0.0.0.0/0
0 0 REJECT all -- virbr0 * 0.0.0.0/0 0.0.0.0/0 reject-with icmp-port-unreachable
Chain LIBVIRT_FWX (1 references)
pkts bytes target prot opt in out source destination
0 0 ACCEPT all -- virbr0 virbr0 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT all -- virbr0 virbr0 0.0.0.0/0 0.0.0.0/0
Chain LIBVIRT_INP (1 references)
pkts bytes target prot opt in out source destination
0 0 ACCEPT udp -- virbr0 * 0.0.0.0/0 0.0.0.0/0 udp dpt:53
0 0 ACCEPT tcp -- virbr0 * 0.0.0.0/0 0.0.0.0/0 tcp dpt:53
0 0 ACCEPT udp -- virbr0 * 0.0.0.0/0 0.0.0.0/0 udp dpt:67
0 0 ACCEPT tcp -- virbr0 * 0.0.0.0/0 0.0.0.0/0 tcp dpt:67
0 0 ACCEPT udp -- virbr0 * 0.0.0.0/0 0.0.0.0/0 udp dpt:53
0 0 ACCEPT tcp -- virbr0 * 0.0.0.0/0 0.0.0.0/0 tcp dpt:53
0 0 ACCEPT udp -- virbr0 * 0.0.0.0/0 0.0.0.0/0 udp dpt:67
0 0 ACCEPT tcp -- virbr0 * 0.0.0.0/0 0.0.0.0/0 tcp dpt:67
Chain LIBVIRT_OUT (1 references)
pkts bytes target prot opt in out source destination
0 0 ACCEPT udp -- * virbr0 0.0.0.0/0 0.0.0.0/0 udp dpt:53
0 0 ACCEPT tcp -- * virbr0 0.0.0.0/0 0.0.0.0/0 tcp dpt:53
0 0 ACCEPT udp -- * virbr0 0.0.0.0/0 0.0.0.0/0 udp dpt:68
0 0 ACCEPT tcp -- * virbr0 0.0.0.0/0 0.0.0.0/0 tcp dpt:68
0 0 ACCEPT udp -- * virbr0 0.0.0.0/0 0.0.0.0/0 udp dpt:53
0 0 ACCEPT tcp -- * virbr0 0.0.0.0/0 0.0.0.0/0 tcp dpt:53
0 0 ACCEPT udp -- * virbr0 0.0.0.0/0 0.0.0.0/0 udp dpt:68
0 0 ACCEPT tcp -- * virbr0 0.0.0.0/0 0.0.0.0/0 tcp dpt:68
iptables -vnL -t nat
Chain PREROUTING (policy ACCEPT 583K packets, 44M bytes)
pkts bytes target prot opt in out source destination
161K 11M CNI-HOSTPORT-DNAT all -- * * 0.0.0.0/0 0.0.0.0/0 ADDRTYPE match dst-type LOCAL
Chain INPUT (policy ACCEPT 579K packets, 43M bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 1163K packets, 73M bytes)
pkts bytes target prot opt in out source destination
1122K 68M CNI-HOSTPORT-DNAT all -- * * 0.0.0.0/0 0.0.0.0/0 ADDRTYPE match dst-type LOCAL
Chain POSTROUTING (policy ACCEPT 1163K packets, 73M bytes)
pkts bytes target prot opt in out source destination
1626 598K MASQUERADE all -- * eth0 10.134.206.0/24 0.0.0.0/0 /* wireguard-nat-rule */
1163K 73M CNI-HOSTPORT-MASQ all -- * * 0.0.0.0/0 0.0.0.0/0 /* CNI portfwd requiring masquerade */
1163K 73M LIBVIRT_PRT all -- * * 0.0.0.0/0 0.0.0.0/0
0 0 CNI-17ab0eb7ca9e577822cf488f all -- * * 10.88.0.3 0.0.0.0/0 /* name: "podman" id: "892b771c0292522a388372805f866e59e6b968a1b08e7631e8ac80e56c5be13f" */
0 0 CNI-00e9b048207815a8f3fc8f27 all -- * * 10.88.0.4 0.0.0.0/0 /* name: "podman" id: "d3bfb858b52a221ad1eecdc402a05c7c8249e5ac006d3339d6ad4d3fdd4251a7" */
2 124 CNI-4c9a18aeb74e61fece7ec177 all -- * * 10.88.0.5 0.0.0.0/0 /* name: "podman" id: "8cfa137dec132f1ce2fb66007ad4b283507f0fbaaf472cc3288e267124b330cb" */
0 0 CNI-aeb527caa7714d83edbf078a all -- * * 10.88.0.23 0.0.0.0/0 /* name: "podman" id: "fcee97ba30df52158c3ce32565c9e74a135e4eb464289dbcd7232f29bde21999" */
0 0 CNI-0327ecd65dfba6f16924cadf all -- * * 10.88.0.3 0.0.0.0/0 /* name: "podman" id: "e6adbe6f6420ff957942f7b112fafadcd66f4d80423b43cdfa1c39ea1d9c4e80" */
0 0 CNI-2882ad3f028617e697dbc2c9 all -- * * 10.88.0.4 0.0.0.0/0 /* name: "podman" id: "dce1a10ee7a1acd4788dcc175f599163c591536f011c3a88b574d7266deaa202" */
0 0 CNI-5cae0369f477326ce9b76e3a all -- * * 10.88.0.5 0.0.0.0/0 /* name: "podman" id: "2125e4265e02677f2a706d1a4c768d3368eed262c102603dcf581b41d1b7dd70" */
0 0 CNI-91d0a321d22c79cbc0ffa33d all -- * * 10.88.0.7 0.0.0.0/0 /* name: "podman" id: "60609096d2c59bf955b340248801ccbb1f173dd6d5eb35ef00defbb9bc26f3db" */
Chain CNI-00e9b048207815a8f3fc8f27 (1 references)
pkts bytes target prot opt in out source destination
0 0 ACCEPT all -- * * 0.0.0.0/0 10.88.0.0/16 /* name: "podman" id: "d3bfb858b52a221ad1eecdc402a05c7c8249e5ac006d3339d6ad4d3fdd4251a7" */
0 0 MASQUERADE all -- * * 0.0.0.0/0 !224.0.0.0/4 /* name: "podman" id: "d3bfb858b52a221ad1eecdc402a05c7c8249e5ac006d3339d6ad4d3fdd4251a7" */
Chain CNI-0327ecd65dfba6f16924cadf (1 references)
pkts bytes target prot opt in out source destination
0 0 ACCEPT all -- * * 0.0.0.0/0 10.88.0.0/16 /* name: "podman" id: "e6adbe6f6420ff957942f7b112fafadcd66f4d80423b43cdfa1c39ea1d9c4e80" */
0 0 MASQUERADE all -- * * 0.0.0.0/0 !224.0.0.0/4 /* name: "podman" id: "e6adbe6f6420ff957942f7b112fafadcd66f4d80423b43cdfa1c39ea1d9c4e80" */
Chain CNI-17ab0eb7ca9e577822cf488f (1 references)
pkts bytes target prot opt in out source destination
0 0 ACCEPT all -- * * 0.0.0.0/0 10.88.0.0/16 /* name: "podman" id: "892b771c0292522a388372805f866e59e6b968a1b08e7631e8ac80e56c5be13f" */
0 0 MASQUERADE all -- * * 0.0.0.0/0 !224.0.0.0/4 /* name: "podman" id: "892b771c0292522a388372805f866e59e6b968a1b08e7631e8ac80e56c5be13f" */
Chain CNI-2882ad3f028617e697dbc2c9 (1 references)
pkts bytes target prot opt in out source destination
0 0 ACCEPT all -- * * 0.0.0.0/0 10.88.0.0/16 /* name: "podman" id: "dce1a10ee7a1acd4788dcc175f599163c591536f011c3a88b574d7266deaa202" */
0 0 MASQUERADE all -- * * 0.0.0.0/0 !224.0.0.0/4 /* name: "podman" id: "dce1a10ee7a1acd4788dcc175f599163c591536f011c3a88b574d7266deaa202" */
Chain CNI-4c9a18aeb74e61fece7ec177 (1 references)
pkts bytes target prot opt in out source destination
0 0 ACCEPT all -- * * 0.0.0.0/0 10.88.0.0/16 /* name: "podman" id: "8cfa137dec132f1ce2fb66007ad4b283507f0fbaaf472cc3288e267124b330cb" */
2 124 MASQUERADE all -- * * 0.0.0.0/0 !224.0.0.0/4 /* name: "podman" id: "8cfa137dec132f1ce2fb66007ad4b283507f0fbaaf472cc3288e267124b330cb" */
Chain CNI-5cae0369f477326ce9b76e3a (1 references)
pkts bytes target prot opt in out source destination
0 0 ACCEPT all -- * * 0.0.0.0/0 10.88.0.0/16 /* name: "podman" id: "2125e4265e02677f2a706d1a4c768d3368eed262c102603dcf581b41d1b7dd70" */
0 0 MASQUERADE all -- * * 0.0.0.0/0 !224.0.0.0/4 /* name: "podman" id: "2125e4265e02677f2a706d1a4c768d3368eed262c102603dcf581b41d1b7dd70" */
Chain CNI-91d0a321d22c79cbc0ffa33d (1 references)
pkts bytes target prot opt in out source destination
0 0 ACCEPT all -- * * 0.0.0.0/0 10.88.0.0/16 /* name: "podman" id: "60609096d2c59bf955b340248801ccbb1f173dd6d5eb35ef00defbb9bc26f3db" */
0 0 MASQUERADE all -- * * 0.0.0.0/0 !224.0.0.0/4 /* name: "podman" id: "60609096d2c59bf955b340248801ccbb1f173dd6d5eb35ef00defbb9bc26f3db" */
Chain CNI-DN-00e9b048207815a8f3fc8 (1 references)
pkts bytes target prot opt in out source destination
0 0 CNI-HOSTPORT-SETMARK tcp -- * * 10.88.0.0/16 0.0.0.0/0 tcp dpt:8999
0 0 CNI-HOSTPORT-SETMARK tcp -- * * 127.0.0.1 0.0.0.0/0 tcp dpt:8999
0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:8999 to:10.88.0.4:3000
Chain CNI-DN-0327ecd65dfba6f16924c (1 references)
pkts bytes target prot opt in out source destination
0 0 CNI-HOSTPORT-SETMARK tcp -- * * 10.88.0.0/16 0.0.0.0/0 tcp dpt:8080
0 0 CNI-HOSTPORT-SETMARK tcp -- * * 127.0.0.1 0.0.0.0/0 tcp dpt:8080
0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:8080 to:10.88.0.3:8080
Chain CNI-DN-17ab0eb7ca9e577822cf4 (1 references)
pkts bytes target prot opt in out source destination
0 0 CNI-HOSTPORT-SETMARK tcp -- * * 10.88.0.0/16 0.0.0.0/0 tcp dpt:1025
0 0 CNI-HOSTPORT-SETMARK tcp -- * * 127.0.0.1 0.0.0.0/0 tcp dpt:1025
0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:1025 to:10.88.0.3:8080
Chain CNI-DN-2882ad3f028617e697dbc (1 references)
pkts bytes target prot opt in out source destination
0 0 CNI-HOSTPORT-SETMARK tcp -- * * 10.88.0.0/16 0.0.0.0/0 tcp dpt:9091
0 0 CNI-HOSTPORT-SETMARK tcp -- * * 127.0.0.1 0.0.0.0/0 tcp dpt:9091
0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:9091 to:10.88.0.4:9090
Chain CNI-DN-4c9a18aeb74e61fece7ec (1 references)
pkts bytes target prot opt in out source destination
0 0 CNI-HOSTPORT-SETMARK tcp -- * * 10.88.0.0/16 0.0.0.0/0 tcp dpt:9091
2 120 CNI-HOSTPORT-SETMARK tcp -- * * 127.0.0.1 0.0.0.0/0 tcp dpt:9091
24 1280 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:9091 to:10.88.0.5:9090
Chain CNI-DN-5cae0369f477326ce9b76 (1 references)
pkts bytes target prot opt in out source destination
0 0 CNI-HOSTPORT-SETMARK tcp -- * * 10.88.0.0/16 0.0.0.0/0 tcp dpt:8999
0 0 CNI-HOSTPORT-SETMARK tcp -- * * 127.0.0.1 0.0.0.0/0 tcp dpt:8999
0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:8999 to:10.88.0.5:3000
Chain CNI-DN-91d0a321d22c79cbc0ffa (1 references)
pkts bytes target prot opt in out source destination
0 0 CNI-HOSTPORT-SETMARK tcp -- * * 10.88.0.0/16 0.0.0.0/0 tcp dpt:1025
0 0 CNI-HOSTPORT-SETMARK tcp -- * * 127.0.0.1 0.0.0.0/0 tcp dpt:1025
0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:1025 to:10.88.0.7:8080
Chain CNI-DN-aeb527caa7714d83edbf0 (1 references)
pkts bytes target prot opt in out source destination
0 0 CNI-HOSTPORT-SETMARK tcp -- * * 10.88.0.0/16 0.0.0.0/0 tcp dpt:8080
0 0 CNI-HOSTPORT-SETMARK tcp -- * * 127.0.0.1 0.0.0.0/0 tcp dpt:8080
10 584 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:8080 to:10.88.0.23:8080
Chain CNI-HOSTPORT-DNAT (2 references)
pkts bytes target prot opt in out source destination
0 0 CNI-DN-17ab0eb7ca9e577822cf4 tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* dnat name: "podman" id: "892b771c0292522a388372805f866e59e6b968a1b08e7631e8ac80e56c5be13f" */ multiport dports 1025
0 0 CNI-DN-00e9b048207815a8f3fc8 tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* dnat name: "podman" id: "d3bfb858b52a221ad1eecdc402a05c7c8249e5ac006d3339d6ad4d3fdd4251a7" */ multiport dports 8999
24 1280 CNI-DN-4c9a18aeb74e61fece7ec tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* dnat name: "podman" id: "8cfa137dec132f1ce2fb66007ad4b283507f0fbaaf472cc3288e267124b330cb" */ multiport dports 9091
10 584 CNI-DN-aeb527caa7714d83edbf0 tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* dnat name: "podman" id: "fcee97ba30df52158c3ce32565c9e74a135e4eb464289dbcd7232f29bde21999" */ multiport dports 8080
0 0 CNI-DN-0327ecd65dfba6f16924c tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* dnat name: "podman" id: "e6adbe6f6420ff957942f7b112fafadcd66f4d80423b43cdfa1c39ea1d9c4e80" */ multiport dports 8080
0 0 CNI-DN-2882ad3f028617e697dbc tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* dnat name: "podman" id: "dce1a10ee7a1acd4788dcc175f599163c591536f011c3a88b574d7266deaa202" */ multiport dports 9091
0 0 CNI-DN-5cae0369f477326ce9b76 tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* dnat name: "podman" id: "2125e4265e02677f2a706d1a4c768d3368eed262c102603dcf581b41d1b7dd70" */ multiport dports 8999
0 0 CNI-DN-91d0a321d22c79cbc0ffa tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* dnat name: "podman" id: "60609096d2c59bf955b340248801ccbb1f173dd6d5eb35ef00defbb9bc26f3db" */ multiport dports 1025
Chain CNI-HOSTPORT-MASQ (1 references)
pkts bytes target prot opt in out source destination
3 160 MASQUERADE all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x2000/0x2000
Chain CNI-HOSTPORT-SETMARK (16 references)
pkts bytes target prot opt in out source destination
2 120 MARK all -- * * 0.0.0.0/0 0.0.0.0/0 /* CNI portfwd masquerade mark */ MARK or 0x2000
Chain CNI-aeb527caa7714d83edbf078a (1 references)
pkts bytes target prot opt in out source destination
0 0 ACCEPT all -- * * 0.0.0.0/0 10.88.0.0/16 /* name: "podman" id: "fcee97ba30df52158c3ce32565c9e74a135e4eb464289dbcd7232f29bde21999" */
0 0 MASQUERADE all -- * * 0.0.0.0/0 !224.0.0.0/4 /* name: "podman" id: "fcee97ba30df52158c3ce32565c9e74a135e4eb464289dbcd7232f29bde21999" */
Chain LIBVIRT_PRT (1 references)
pkts bytes target prot opt in out source destination
620 81523 RETURN all -- * * 192.168.122.0/24 224.0.0.0/24
0 0 RETURN all -- * * 192.168.122.0/24 255.255.255.255
0 0 MASQUERADE tcp -- * * 192.168.122.0/24 !192.168.122.0/24 masq ports: 1024-65535
17 10795 MASQUERADE udp -- * * 192.168.122.0/24 !192.168.122.0/24 masq ports: 1024-65535
0 0 MASQUERADE all -- * * 192.168.122.0/24 !192.168.122.0/24
0 0 RETURN all -- * * 192.168.122.0/24 224.0.0.0/24
0 0 RETURN all -- * * 192.168.122.0/24 255.255.255.255
0 0 MASQUERADE tcp -- * * 192.168.122.0/24 !192.168.122.0/24 masq ports: 1024-65535
0 0 MASQUERADE udp -- * * 192.168.122.0/24 !192.168.122.0/24 masq ports: 1024-65535
0 0 MASQUERADE all -- * * 192.168.122.0/24 !192.168.122.0/24
slavid commented
A very similar issue happens using Docker instead of Podman.
If you configure a Docker (such as b4bz/homer that I use myself too) with a reverse proxy (in my case traefik) and there was a previous installation of PiVPN on the system, then traefik just doesn't work (probably due to a conflict with the IPTABLES configured by PiVPN).
My fix was to uninstall PiVPN and then reinstall it. If you install PiVPN AFTER the Dockers are already working then everything runs fine even after restarting but if you do it the other way by installing first PiVPN then Docker and finally run the docker containers it won't work.
To make it persistent after reboots edit the file /lib/systemd/system/wg-quick@.service (for wireguard) and add docker.socket docker.service in the line After= so that the VPN starts after the Docker service:
After=network-online.target nss-lookup.target docker.socket docker.service
Probably it is what this solution on stackoverflow says.