Flatcar with container openvpn-client not wokring properly
Opened this issue · 11 comments
Description
Flatcar with openvpn client container not wokring
Impact
Unable to use flatcar with openvpn
Environment and steps to reproduce
- Set-up:
Flatcar image: flatcar_production_openstack_image.img
pool-g4dzrku5-sj3dtqihuu6cjof6-node-hshs3fbx ~ # cat /etc/os-release
NAME="Flatcar Container Linux by Kinvolk"
ID=flatcar
ID_LIKE=coreos
VERSION=3975.2.0
VERSION_ID=3975.2.0
BUILD_ID=2024-08-05-2103
SYSEXT_LEVEL=1.0
PRETTY_NAME="Flatcar Container Linux by Kinvolk 3975.2.0 (Oklo)"
ANSI_COLOR="38;5;75"
HOME_URL="https://flatcar.org/"
BUG_REPORT_URL="https://issues.flatcar.org"
FLATCAR_BOARD="amd64-usr"
CPE_NAME="cpe:2.3:o:flatcar-linux:flatcar_linux:3975.2.0:*:*:*:*:*:*:*"
- Task: Running Openvpn-client container (I have tried a lot docker openvpn-client on github and build one myself)
2.1: Install docker-compose
curl -SL https://github.com/docker/compose/releases/download/v2.29.1/docker-compose-linux-x86_64 -o /opt/bin/docker-compose
chmod +x /opt/bin/docker-compose
2.2: Write docker-compose VPN file
version: "3.3"
services:
vpn:
image: cr-hn-1.bizflycloud.vn/31ff9581861a4d0ea4df5e7dda0f665d/openvpn-client@sha256:2defe3062e65ad0ecf43bc747d60d1274ebbf7a45100c5cf5448fd7114caac80
volumes:
- /etc/openvpn/kengine.conf:/vpn/kengine.conf
- /dev/net/tun:/dev/net/tun
cap_add:
- NET_ADMIN
restart: always
network_mode: "host"
2.3: Write openvpn-client file
client
dev kengine
dev-type tap
reneg-sec 0
proto tcp-client
remote xxx.xx.xx.xx xxxxx
resolv-retry infinite
nobind
<ca>
-----BEGIN CERTIFICATE-----
.........
-----END CERTIFICATE-----
</ca>
<key>
-----BEGIN PRIVATE KEY-----
..........
-----END PRIVATE KEY-----
</key>
<cert>
-----BEGIN CERTIFICATE-----
......
-----END CERTIFICATE-----
</cert>
<tls-auth>
#
# 2048 bit OpenVPN static key
#
-----BEGIN OpenVPN Static key V1-----
...........
-----END OpenVPN Static key V1-----
</tls-auth>
remote-cert-tls server
key-direction 1
script-security 3
keepalive 10 60
persist-key
persist-tun
comp-lzo
verb 3
- Action(s):
a. Run the docker-compose
pool-g4dzrku5-sj3dtqihuu6cjof6-node-hshs3fbx ~ # docker-compose -f docker-compose-2.yaml up
WARN[0000] /root/docker-compose-2.yaml: the attribute `version` is obsolete, it will be ignored, please remove it to avoid potential confusion
WARN[0000] Found orphan containers ([openvpn-client root-openvpn-1]) for this project. If you removed or renamed this service in your compose file, you can run this command with the --remove-orphans flag to clean it up.
[+] Running 1/0
✔ Container root-vpn-1 Created 0.0s
Attaching to vpn-1
vpn-1 | + exec sg vpn -c 'openvpn --cd /vpn --config /vpn/kengine.conf --script-security 2 --redirect-gateway def1 '
vpn-1 | Mon Aug 12 08:30:58 2024 OpenVPN 2.4.9 x86_64-alpine-linux-musl [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [MH/PKTINFO] [AEAD] built on Apr 20 2020
vpn-1 | Mon Aug 12 08:30:58 2024 library versions: OpenSSL 1.1.1g 21 Apr 2020, LZO 2.10
vpn-1 | Mon Aug 12 08:30:58 2024 Outgoing Control Channel Authentication: Using 160 bit message hash 'SHA1' for HMAC authentication
vpn-1 | Mon Aug 12 08:30:58 2024 Incoming Control Channel Authentication: Using 160 bit message hash 'SHA1' for HMAC authentication
vpn-1 | Mon Aug 12 08:30:58 2024 TCP/UDP: Preserving recently used remote address: [AF_INET]123.31.11.151:10001
vpn-1 | Mon Aug 12 08:30:58 2024 Socket Buffers: R=[131072->131072] S=[16384->16384]
vpn-1 | Mon Aug 12 08:30:58 2024 Attempting to establish TCP connection with [AF_INET]123.31.11.151:10001 [nonblock]
vpn-1 | Mon Aug 12 08:30:59 2024 TCP connection established with [AF_INET]123.31.11.151:10001
vpn-1 | Mon Aug 12 08:30:59 2024 TCP_CLIENT link local: (not bound)
vpn-1 | Mon Aug 12 08:30:59 2024 TCP_CLIENT link remote: [AF_INET]123.31.11.151:10001
vpn-1 | Mon Aug 12 08:30:59 2024 TLS: Initial packet from [AF_INET]123.31.11.151:10001, sid=b30c806f 6b1db9ce
vpn-1 | Mon Aug 12 08:30:59 2024 VERIFY OK: depth=1, CN=bke-vpn
vpn-1 | Mon Aug 12 08:30:59 2024 VERIFY KU OK
vpn-1 | Mon Aug 12 08:30:59 2024 Validating certificate extended key usage
vpn-1 | Mon Aug 12 08:30:59 2024 ++ Certificate has EKU (str) TLS Web Server Authentication, expects TLS Web Server Authentication
vpn-1 | Mon Aug 12 08:30:59 2024 VERIFY EKU OK
vpn-1 | Mon Aug 12 08:30:59 2024 VERIFY OK: depth=0, CN=bke-vpn
vpn-1 | Mon Aug 12 08:30:59 2024 Control Channel: TLSv1.3, cipher TLSv1.3 TLS_AES_256_GCM_SHA384, 2048 bit RSA
vpn-1 | Mon Aug 12 08:30:59 2024 [bke-vpn] Peer Connection Initiated with [AF_INET]123.31.11.151:10001
vpn-1 | Mon Aug 12 08:31:00 2024 SENT CONTROL [bke-vpn]: 'PUSH_REQUEST' (status=1)
vpn-1 | Mon Aug 12 08:31:00 2024 PUSH: Received control message: 'PUSH_REPLY,route-gateway 10.99.0.1,ping 20,ping-restart 60,ifconfig 10.99.0.4 255.255.240.0,peer-id 0,cipher AES-256-GCM'
vpn-1 | Mon Aug 12 08:31:00 2024 OPTIONS IMPORT: timers and/or timeouts modified
vpn-1 | Mon Aug 12 08:31:00 2024 OPTIONS IMPORT: --ifconfig/up options modified
vpn-1 | Mon Aug 12 08:31:00 2024 OPTIONS IMPORT: route-related options modified
vpn-1 | Mon Aug 12 08:31:00 2024 OPTIONS IMPORT: peer-id set
vpn-1 | Mon Aug 12 08:31:00 2024 OPTIONS IMPORT: adjusting link_mtu to 1659
vpn-1 | Mon Aug 12 08:31:00 2024 OPTIONS IMPORT: data channel crypto options modified
vpn-1 | Mon Aug 12 08:31:00 2024 Data Channel: using negotiated cipher 'AES-256-GCM'
vpn-1 | Mon Aug 12 08:31:00 2024 Outgoing Data Channel: Cipher 'AES-256-GCM' initialized with 256 bit key
vpn-1 | Mon Aug 12 08:31:00 2024 Incoming Data Channel: Cipher 'AES-256-GCM' initialized with 256 bit key
vpn-1 | Mon Aug 12 08:31:00 2024 ROUTE_GATEWAY 103.107.182.1/255.255.255.0 IFACE=eth0 HWADDR=fa:16:3e:c1:ee:2b
vpn-1 | Mon Aug 12 08:31:00 2024 TUN/TAP device kengine opened
vpn-1 | Mon Aug 12 08:31:00 2024 TUN/TAP TX queue length set to 100
vpn-1 | Mon Aug 12 08:31:00 2024 /sbin/ip link set dev kengine up mtu 1500
vpn-1 | Mon Aug 12 08:31:00 2024 /sbin/ip addr add dev kengine 10.99.0.4/20 broadcast 10.99.15.255
vpn-1 | Mon Aug 12 08:31:00 2024 /sbin/ip route add 123.31.11.151/32 via 103.107.182.1
vpn-1 | Mon Aug 12 08:31:00 2024 /sbin/ip route add 0.0.0.0/1 via 10.99.0.1
vpn-1 | Mon Aug 12 08:31:00 2024 /sbin/ip route add 128.0.0.0/1 via 10.99.0.1
vpn-1 | Mon Aug 12 08:31:00 2024 WARNING: this configuration may cache passwords in memory -- use the auth-nocache option to prevent this
vpn-1 | Mon Aug 12 08:31:00 2024 Initialization Sequence Completed
This seem working but actually not, the ip addr add dev kengine 10.99.0.4/20 broadcast 10.99.15.255
is not working so there is no ip on the kengine
112: kengine: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN group default qlen 100
link/ether 52:a3:e8:10:4c:96 brd ff:ff:ff:ff:ff:ff
inet6 fe80::ac5b:61ff:fe54:98f6/64 scope link proto kernel_ll
valid_lft forever preferred_lft forever
b. I have to manualy add myself and the VPN finally work
112: kengine: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN group default qlen 100
link/ether 52:a3:e8:10:4c:96 brd ff:ff:ff:ff:ff:ff
inet 10.99.0.4/20 brd 10.99.15.255 scope global kengine
valid_lft forever preferred_lft forever
inet6 fe80::ac5b:61ff:fe54:98f6/64 scope link proto kernel_ll
valid_lft forever preferred_lft forever
Testing
pool-g4dzrku5-sj3dtqihuu6cjof6-node-hshs3fbx ~ # ping 10.99.0.1
PING 10.99.0.1 (10.99.0.1) 56(84) bytes of data.
64 bytes from 10.99.0.1: icmp_seq=1 ttl=64 time=2.50 ms
64 bytes from 10.99.0.1: icmp_seq=2 ttl=64 time=1.42 ms
64 bytes from 10.99.0.1: icmp_seq=3 ttl=64 time=1.45 ms
^C
--- 10.99.0.1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 1.416/1.786/2.496/0.502 ms
pool-g4dzrku5-sj3dtqihuu6cjof6-node-hshs3fbx ~ # ping 10.99.0.2
PING 10.99.0.2 (10.99.0.2) 56(84) bytes of data.
64 bytes from 10.99.0.2: icmp_seq=1 ttl=64 time=2.11 ms
64 bytes from 10.99.0.2: icmp_seq=2 ttl=64 time=2.75 ms
64 bytes from 10.99.0.2: icmp_seq=3 ttl=64 time=2.14 ms
^C
--- 10.99.0.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 2.109/2.334/2.752/0.295 ms
but since i want automaticly run VPN and I cant add it manual all the time so this problem need look up to
4. Error:
Not add IP properly on VPN interface
Expected behavior
The IP on interface automaticlly added
Additional information
None
Can you share the output of networkctl status
from the host? Does the device show up as unmanaged?
Are you able to create a minimal reproducer for this?
Can you share the output of
networkctl status
from the host? Does the device show up as unmanaged?Are you able to create a minimal reproducer for this?
pool-g4dzrku5-sj3dtqihuu6cjof6-node-hshs3fbx ~ # networkctl status
● Interfaces: 1, 2, 3, 7, 8, 12, 14, 16, 24, 20, 22, 18, 30, 28, 26, 33, 34, 4, 113
State: routable
Online state: online
Address: 103.107.182.222 on eth0
10.20.4.127 on eth1
172.17.0.1 on docker0
10.200.0.159 on cilium_host
172.18.0.1 on br-098937d94946
fe80::f816:3eff:fec1:ee2b on eth0
fe80::f816:3eff:fefd:b77d on eth1
fe80::42:96ff:fe5b:9fc4 on docker0
fe80::40d7:33ff:fe7a:910b on cilium_net
fe80::48a5:55ff:fe9d:3bb2 on cilium_host
fe80::e037:abff:fe5c:6eb3 on lxc99eb9c4d3eb4
fe80::6c06:47ff:fe44:2265 on lxcd81121c9fde0
fe80::90dc:a0ff:fe41:bea0 on lxce0fba8614383
fe80::b842:75ff:fee7:cec3 on lxc6c022ee53b5c
fe80::40f3:58ff:fe97:219a on lxc21e65491adf6
fe80::a448:14ff:fe58:3189 on lxc7df0526efc64
fe80::1070:18ff:fe05:e8c0 on lxc59b2200808e0
fe80::6c7e:31ff:fe1f:316d on lxcc625a36d34a6
fe80::5c8e:d2ff:fe18:106 on lxc6316b6423586
fe80::8463:c4ff:fe74:5fb1 on lxcb04e3a7afb05
fe80::2cc2:e2ff:fe77:6d04 on lxc_health
fe80::42:97ff:fe94:1333 on br-098937d94946
fe80::d489:f4ff:fe2a:18e4 on kengine
Gateway: 103.107.182.1 on eth0
DNS: 8.8.8.8
45.124.93.62
Search Domains: openstacklocal
Aug 12 08:31:00 pool-g4dzrku5-sj3dtqihuu6cjof6-node-hshs3fbx systemd-networkd[1169]: kengine: Gained carrier
Aug 12 08:31:00 pool-g4dzrku5-sj3dtqihuu6cjof6-node-hshs3fbx systemd-networkd[1169]: kengine: Configuring with /usr/lib/systemd/network/zz-default.network.
Aug 12 08:31:02 pool-g4dzrku5-sj3dtqihuu6cjof6-node-hshs3fbx systemd-networkd[1169]: kengine: Gained IPv6LL
Aug 12 09:46:24 pool-g4dzrku5-sj3dtqihuu6cjof6-node-hshs3fbx systemd-networkd[1169]: kengine: Link DOWN
Aug 12 09:46:24 pool-g4dzrku5-sj3dtqihuu6cjof6-node-hshs3fbx systemd-networkd[1169]: kengine: Lost carrier
Aug 12 09:46:24 pool-g4dzrku5-sj3dtqihuu6cjof6-node-hshs3fbx systemd-networkd[1169]: kengine: DHCPv6 lease lost
Aug 12 09:46:30 pool-g4dzrku5-sj3dtqihuu6cjof6-node-hshs3fbx systemd-networkd[1169]: kengine: Link UP
Aug 12 09:46:30 pool-g4dzrku5-sj3dtqihuu6cjof6-node-hshs3fbx systemd-networkd[1169]: kengine: Gained carrier
Aug 12 09:46:30 pool-g4dzrku5-sj3dtqihuu6cjof6-node-hshs3fbx systemd-networkd[1169]: kengine: Configuring with /usr/lib/systemd/network/zz-default.network.
Aug 12 09:46:31 pool-g4dzrku5-sj3dtqihuu6cjof6-node-hshs3fbx systemd-networkd[1169]: kengine: Gained IPv6LL
here
I can give you the client full config if you want
I have tried other openvpn-client and build 1 myself but seem they have common error not adding ip on itnerface
sorry, i meant networkctl list
but this already confirms my suspicion that networkd is trying to manage your device:
systemd-networkd[1169]: kengine: Configuring with /usr/lib/systemd/network/zz-default.network.
but this already confirms my suspicion that networkd is trying to manage your device:
systemd-networkd[1169]: kengine: Configuring with /usr/lib/systemd/network/zz-default.network.
IDX LINK TYPE OPERATIONAL SETUP
1 lo loopback carrier unmanaged
2 eth0 ether routable configured
3 eth1 ether routable configured
4 docker0 bridge no-carrier unmanaged
7 cilium_net ether degraded unmanaged
8 cilium_host ether routable unmanaged
12 lxc99eb9c4d3eb4 ether degraded unmanaged
14 lxcd81121c9fde0 ether degraded unmanaged
16 lxce0fba8614383 ether degraded unmanaged
18 lxc6c022ee53b5c ether degraded unmanaged
20 lxc21e65491adf6 ether degraded unmanaged
22 lxc7df0526efc64 ether degraded unmanaged
24 lxc59b2200808e0 ether degraded unmanaged
26 lxcc625a36d34a6 ether degraded unmanaged
28 lxc6316b6423586 ether degraded unmanaged
30 lxcb04e3a7afb05 ether degraded unmanaged
33 lxc_health ether degraded unmanaged
34 br-098937d94946 bridge no-carrier unmanaged
113 kengine ether degraded configuring
19 links listed.
look like is not working properly, so how to fix this problem
for a quick fix create /etc/systemd/network/kengine.network
with these contents:
[Match]
Name=kengine
[Link]
Unmanaged=yes
and then systemctl restart systemd-networkd
.
for a quick fix create
/etc/systemd/network/kengine.network
with these contents:[Match] Name=kengine [Link] Unmanaged=yes
and then
systemctl restart systemd-networkd
.
wow ok it worked for now, tks for your support
I'll leave this issue open as we should exclude tun/tap interfaces from networkd management by default.
@jepio Hey Jeremi! How are you doing? I'm thinking of adding a new configuration file with the following content:
[Match]
Kind=tun tap
[Link]
Unmanaged=yes
This configuration will match all tun
and tap
interfaces and set them as unmanaged. I can make a PR if this looks good to you :)
Sure, try it out. The PR needs to be against flatcar/init and flatcar/bootengine (for consistency).
Sure, try it out. The PR needs to be against flatcar/init and flatcar/bootengine (for consistency).
Cool, I started a discussion here: link