SUSE-Enceladus/cloud-netconfig

The reverse path filtering takes a place on network interfaces when the host has other interface in the same subnet as destination.

andriysgit opened this issue · 0 comments

andriysgit commented

This was tested in AWS with SLES12 (latest cloud-netconfig-ec2-1.6-24.1.noarch) / SELS15 (latest cloud-netconfig-ec2-1.6-25.5.1.noarch).
The issue: cloud-netconfig adds subnet with interface found on the host, into routing tables of each other interface. This causes traffic from network interface to be routed to leave from another interface which is in the same subnet as destination, causing it being dropped by kernel due to reverse path filtering.

Scenario:

  1. Instance A and B have eth0s in the same subnet 172.31.0.0/20. (A eth0: 172.31.15.200/20; B eth0 172.31.11.232/20)
  2. Instance B has eth1 interface in another subnet 172.31.128.0/24 (B eth1: 172.31.128.22/24)
  3. Communication between eth0s of A and B works.
  4. Instance A can't reach out to eth1 on B. B can't reach from eth1 to eth0 on A .

Then it can be observed that cloud-netconfig sets routing tables per interfaces on B as bellow with subnets of other interfaces added:

# ip route show table 100
default via 172.31.0.1 dev eth0
172.31.0.0/20 dev eth0 proto kernel scope link src 172.31.11.151
172.31.128.0/24 dev eth1 proto kernel scope link src 172.31.128.22
# ip route show table 101
default via 172.31.128.1 dev eth1
172.31.0.0/20 dev eth0 proto kernel scope link src 172.31.11.151
172.31.128.0/24 dev eth1 proto kernel scope link src 172.31.128.22

Bellow shows that traffic from eth1 on B to eth0 on A will be routed through eth0 on B:

# ip route get from 172.31.128.22 172.31.15.200
172.31.15.200 from 172.31.128.22 dev eth0 table 101 uid 0

The reverse path filtering takes a place and connections are timed out.

For a very short period of time this can be addressed by manually removing route to eth0 from RT of eth1:

# ip route del 172.31.0.0/20 dev eth0 proto kernel scope link src 172.31.11.151 table 101
# ip route show table 101
default via 172.31.128.1 dev eth1
172.31.128.0/24 dev eth1 proto kernel scope link src 172.31.128.22
# ip route get from 172.31.128.22 172.31.15.200
172.31.15.200 from 172.31.128.22 via 172.31.128.1 dev eth1 table 101 uid 0

The permanent workaround is to make a change in /etc/netconfig.d/cloud-netconfig to avoid subnets of others network interfaces to be added into routing table of each interface.
from:
ip $ipv route show dev "$iface" | grep -v "^default" | while read route ; do
to
ip $ipv route show dev "$iface" | grep -v "^default" | grep $iface | while read route ; do

In the following section of cloud-netconfig:

# -------------------------------------------------------------------
# copy routes from default table to given table and remove routes
# that do not exist in default table (if any)
#
update_routing_table()
{
    local rtable="$1"
    local iface="$2"

    test -z "$rtable" && return 1

    # copy routes from default table
    for ipv in "-4" "-6" ; do
        ip $ipv route show | grep -v "^default" | while read route ; do
            ip $ipv route replace $route table $rtable
        done

        # Do a second run with routes specific to the interface being configured.
        # This makes sure that in case there are competing routes (same destination
        # but different device), the one matching the right interface is selected.
        # This situation can happen in case there are multiple interfaces that are
        # connected to the same subnet.
        ip $ipv route show dev "$iface" | grep -v "^default" | while read route ; do
            ip $ipv route replace $route dev "$iface" table $rtable
        done

        # check if there are any leftover routes and delete them
        ip $ipv route show all table $rtable | grep -v "^default" | while read route ; do
            ip_out="$(ip $ipv route show $route)"
            test -z "$ip_out" && ip $ipv route del $route table $rtable
        done
    done
}

# -------------------------------------------------------------------

Then restart wicked:

# systemctl restart wicked
# ip route show table 101
default via 172.31.128.1 dev eth1
172.31.128.0/24 dev eth1 proto kernel scope link src 172.31.128.22
# ping -I eth1 172.31.15.200
PING 172.31.15.200 (172.31.15.200) from 172.31.128.22 eth1: 56(84) bytes of data.
64 bytes from 172.31.15.200: icmp_seq=1 ttl=255 time=0.362 ms
..

additional details:
### SLES 12sp5:

# cat /etc/os-release
NAME="SLES"
VERSION="12-SP5"
VERSION_ID="12.5"
PRETTY_NAME="SUSE Linux Enterprise Server 12 SP5"
..
# rpm -qf /etc/netconfig.d/cloud-netconfig
# cloud-netconfig-ec2-1.6-24.1.noarch
ip-172-31-8-16:/home/ec2-user # zypper info cloud-netconfig-ec2
Information for package cloud-netconfig-ec2:
--------------------------------------------
Repository     : SLE-Module-Public-Cloud12-Updates
Name           : cloud-netconfig-ec2
Version        : 1.6-24.1
Arch           : noarch
Vendor         : SUSE LLC <https://www.suse.com/>
Support Level  : Level 3
Installed Size : 62.7 KiB
Installed      : Yes
Status         : up-to-date
Source package : cloud-netconfig-ec2-1.6-24.1.src
Summary        : Network configuration scripts for Amazon EC2
Description    :
    This package contains scripts for automatically configuring network interfaces
    in Amazon EC2 with full support for hotplug.

###SLES 15sp2:

# cat /etc/os-release
NAME="SLES"
VERSION="15-SP2"
VERSION_ID="15.2"
PRETTY_NAME="SUSE Linux Enterprise Server 15 SP2"
..
# rpm -qf /etc/netconfig.d/cloud-netconfig
cloud-netconfig-ec2-1.6-25.5.1.noarch
# zypper info cloud-netconfig-ec2
..
Information for package cloud-netconfig-ec2:
--------------------------------------------
Repository     : SLE-Module-Public-Cloud15-SP2-Updates
Name           : cloud-netconfig-ec2
Version        : 1.6-25.5.1
Arch           : noarch
Vendor         : SUSE LLC <https://www.suse.com/>
Support Level  : Level 3
Installed Size : 62.7 KiB
Installed      : Yes
Status         : up-to-date
Source package : cloud-netconfig-ec2-1.6-25.5.1.src
Summary        : Network configuration scripts for Amazon EC2
Description    :
    This package contains scripts for automatically configuring network interfaces
    in Amazon EC2 with full support for hotplug.