containers with Container Network static IP are not accesible from netowk after upgrading VCH to 1.5.2
qxmips opened this issue · 5 comments
Summary
With VCH 1.5.2 when a container connected to both an external container network (vlan72-vic-containers) and the bridge network, it is not available from the container network.
vic-prod@rp01:/opt/prod/sso$ docker network ls
NETWORK ID NAME DRIVER SCOPE
a864ecbf0a6a bridge bridge
f45dc71d666e vlan72-vic-containers external
ARP table shows MAC address from the bridge interface of the container against the container network IP, not the MAC address of container interface that is connected to external
'container network'
contaier network interfces:
root@smb:/# ip a s
1: lo: <LOOPBACK,UP,LOWER_UP> .....
2: eth0: 00:50:56:aa:6e:b1 inet 10.4.72.151/24
3: eth1: 00:50:56:aa:00:d0 inet 172.20.0.6/16
ROUTER ARP:
show arp
10.4.72.151 ether 00:50:56:aa:00:d0 C eth4 <-- MAC of eth1 , should be eth0
with 1.4.4 works as expected
Environment information
vSphere and vCenter Server version
6.7.0 build 13007421
VIC version
v1.5.2.1500
VCH installer version v1.5.2-20879-30b67a14
VCH configuration
inspect VCH cofig info:
INFO[0000] The target VCH is configured with the following options:
--target=https://*******
--thumbprint=******
--name=vch-prod
--compute-resource=******
--ops-user=****@vsphere.local
--image-store=ds://****
--container-name-convention={name}-prod
--volume-store=ds://*****/VIC:default
--volume-store=ds://*****/VIC:san-prod-02
--volume-store=ds://*****/VIC:san-prod-03
--dns-server=*****
--bridge-network=vlan72-vic-bridge
--bridge-network-range=172.20.0.0/16
--public-network=vlan72-10.4.72-WebApps
--public-network-gateway=10.4.72.254
--public-network-ip=10.4.72.10/24
--container-network=vlan72-vic-containers:vlan72-vic-containers
--container-network-gateway=vlan72-vic-containers:10.4.72.254/24
--container-network-ip-range=vlan72-vic-containers:10.4.72.151-10.4.72.199
--container-network-dns=vlan72-vic-containers:10.4.34.80
--container-network-dns=vlan72-vic-containers:10.4.34.81
--container-network-firewall=vlan72-vic-containers:open
--syslog-address=tcp://******l:5140
Details
Steps to reproduce
We use production containers connected to a contaner network with static ip configured
vic-prod@rp01:/opt/prod/sso$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
....
fe45af196bc2 ubuntu "bash" 11 minutes ago Up 11 minutes 0/* ubuntu
59ac85ed2b58 vic01-san.ztelco.local/prod/sso:latest "docker-php-entrypoi…" 12 minutes ago Up 12 minutes 10.4.72.151:0->0/*, 10.4.72.151:8081->80/tcp sso
container is connected to both bridge and container expernal network:
vic-prod@rp01:/opt/prod/sso$ cat docker-compose-vic-production.yml
version: '3'
services:
app:
hostname: smb
networks:
vlan72-vic-containers:
ipv4_address: ${SSO_IP}
bridge:
networks:
vlan72-vic-containers:
external: true
bridge:
external: true
after upgrading the VCH to 1.5.2 and recreating the containers, containers are not accessible from the network:
vic-prod@rp01:/opt/prod/sso$ ping 10.4.72.151
PING 10.4.72.151 (10.4.72.151) 56(84) bytes of data.
...
when I login to the network router I can see that the container static IP address is shown on ARP table with the MAC address of the container bridge interface
ROUTER:
vyos@rp-mgmt-rtr01:~$ show arp |grep 72
...
10.4.72.151 ether 00:50:56:aa:00:d0 C eth4
inside container:
root@smb:/# ip a s
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 00:50:56:aa:6e:b1 brd ff:ff:ff:ff:ff:ff
inet 10.4.72.151/24 scope global eth0
valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 00:50:56:aa:00:d0 brd ff:ff:ff:ff:ff:ff
inet 172.20.0.6/16 scope global eth1
valid_lft forever preferred_lft forever
when pinging the gateway from container it becomes availabe for a while
root@smb:/# ping 10.4.72.254
PING 10.4.72.254 (10.4.72.254) 56(84) bytes of data.
64 bytes from 10.4.72.254: icmp_seq=1 ttl=64 time=1.31 ms
64 bytes from 10.4.72.254: icmp_seq=2 ttl=64 time=0.270 ms
ROUTER:
qxmips@rp01:~$ ping 10.4.72.151
PING 10.4.72.151 (10.4.72.151) 56(84) bytes of data.
64 bytes from 10.4.72.151: icmp_seq=1 ttl=62 time=0.725 ms
....
vyos@rp-mgmt-rtr01:~$ show arp |grep 72
10.4.72.151 ether 00:50:56:aa:6e:b1 C eth4 <- !!!NOTE the MAC ADDRESS has changed
AFTER few mins:
vic-prod@rp01:/opt/prod/sso$ ping 10.4.72.151
PING 10.4.72.151 (10.4.72.151) 56(84) bytes of data.
vyos@rp-mgmt-rtr01:~$ show arp |grep 72
...
10.4.72.151 ether 00:50:56:aa:00:d0 C eth4
Actual behavior
a container with is connected to both bridge and external is not available by external ip
Expected behavior
the container is available by external IP
Troubleshooting attempted
redeployed VCH
@yuyangbj We also came across similar issue after VCH upgrade from 1.4.3 to 1.5.3. Not sure if both issues are related.
VCH is created with Static IP Range (Container Network Range parameter).
Error Message: ERROR Handler for POST /v1.25/containers/"containerid"/start returned error: Server error from portlayer: Cannot reserve IP range "Ip address" - "Ip address". Already in use
We are able to run container with static ip address range after reverting back to 1.4.3 VCH version,
Kindly let us know if any additional details are needed and both issues are not related will raise a different one.
@aviratna can you tell me how to reproduce this issue? From VIC 1.5.2, we will never release ip address until the container is deleted.
Please find the steps below:
- Create a VCH with version 1.4.3 with container network without DHCP support.
- Use --cnr to specify IP Address range.
- Create containers using vch endpoint, containers will get ip address from static ip address range specified during VCH creation.
- Upgrade VCH from 1.4.3 to 1.5.3
- Try new container creation, ip address will not get allocated
- VCH logs will show below error "Cannot reserve IP range "Ip address" - "Ip address". Already in use".
This issue happens only for VCH which are created with container network without DHCP support.
Upgrade from 1.4.3 to 1.5.3 works fine for VCH which are created using container network which supports DHCP.