gateway is set only if a 2nd route is defined
kisahm opened this issue · 0 comments
kisahm commented
Hi,
i'm using Whereabouts to manage the ipam for kubevirt.
Env:
- Ubuntu 20.04
- Kubernetes v1.24.6
- Whereabouts v0.6
- Kubevirt v0.58.0
cni type "bridge" will be used
bridge br0 is connected to physical interface, vlan_filtering is enabled: ip link set br0 type bridge vlan_filtering 1
PROBLEM
If gateway and / or route for 0.0.0.0/0 is set, the default gateway will no be announced to the virtual machine.
The dns settings are ignored as well
NetworkAttachmentDefinition:
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
name: br-vlan-200
spec:
config: '{
"cniVersion": "0.3.1",
"type": "bridge",
"bridge": "br0",
"vlan": 200,
"ipam": {
"type": "whereabouts",
"range": "10.250.200.10-10.250.200.250/24",
"gateway": "10.250.200.1",
"log_level": "debug",
"dns": {
"nameservers":[
"8.8.8.8",
"1.1.1.1"
]
},
"routes": [
{
"dst": "0.0.0.0/0",
"gw": "10.250.200.1"
}
]
}
}'
VM definition:
apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
name: vlan200-vm1
spec:
running: true
template:
metadata:
labels:
kubevirt.io/size: small
kubevirt.io/domain: 200100-vm1
spec:
domain:
devices:
disks:
- name: containerdisk
disk:
bus: virtio
- name: cloudinitdisk
disk:
bus: virtio
interfaces:
- name: br-vlan-200
bridge: {}
resources:
requests:
memory: 128M
networks:
- name: br-vlan-200
multus:
networkName: br-vlan-200
volumes:
- name: containerdisk
containerDisk:
image: quay.io/kubevirt/cirros-container-disk-demo
- name: cloudinitdisk
cloudInitNoCloud:
userDataBase64: SGkuXG4=
on the vm:
login as 'cirros' user. default password: 'gocubsgo'. use 'sudo' for root.
vlan200-vm1 login: cirros
Password:
$ ip route
10.250.200.0/24 dev eth0 src 10.250.200.10
$ cat /etc/resolv.conf
search default.svc.cluster.local
nameserver 10.96.0.10
WORKAROUND:
add 2nd route.
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
name: br-vlan-200
spec:
config: '{
"cniVersion": "0.3.1",
"type": "bridge",
"bridge": "br0",
"vlan": 200,
"ipam": {
"type": "whereabouts",
"range": "10.250.200.10-10.250.200.250/24",
"gateway": "10.250.200.1",
"log_level": "debug",
"dns": {
"nameservers":[
"8.8.8.8",
"1.1.1.1"
]
},
"routes": [
{
"dst": "0.0.0.0/0",
"gw": "10.250.200.1"
},
{
"dst": "10.250.0.0/24",
"gw": "10.250.200.1"
}
]
}
}'
on the vm:
login as 'cirros' user. default password: 'gocubsgo'. use 'sudo' for root.
vlan200-vm1 login: cirros
Password:
$ ip route
default via 10.250.200.1 dev eth0
10.250.0.0/24 via 10.250.200.1 dev eth0
10.250.200.0/24 dev eth0 src 10.250.200.10
$ cat /etc/resolv.conf
search default.svc.cluster.local
nameserver 10.96.0.10
With a 2nd route the gateway is available. why?
dns nameservers are ignored as well. didn't find a workaround atm.
whereabouts log:
2023-01-03T09:24:28Z [verbose] pod controller created
2023-01-03T09:24:28Z [verbose] Starting informer factories ...
2023-01-03T09:24:28Z [verbose] Informer factories started
2023-01-03T09:24:28Z [verbose] starting network controller
2023-01-03T09:32:19Z [verbose] deleted pod [default/virt-launcher-vlan200-vm1-2tmv7]
2023-01-03T09:32:19Z [verbose] skipped net-attach-def for default network
2023-01-03T09:32:19Z [debug] pod's network status: {Name:default/br-vlan-200 Interface:net1 IPs:[10.250.200.10] Mac:ba:b2:77:2a:df:ee Default:false DNS:{Nameservers:[] Domain: Search:[] Options:[]} DeviceInfo:<nil>}
2023-01-03T09:32:19Z [verbose] the NAD's config: {{ "cniVersion": "0.3.1", "type": "bridge", "bridge": "br0", "vlan": 200, "ipam": { "type": "whereabouts", "range": "10.250.200.10-10.250.200.250/24", "gateway": "10.250.200.1", "log_level": "debug", "dns": { "nameservers":[ "8.8.8.8", "1.1.1.1" ] }, "routes": [ { "dst": "0.0.0.0/0", "gw": "10.250.200.1" } ] } }}
2023-01-03T09:32:19Z [debug] Used defaults from parsed flat file config @ /host/etc/cni/net.d/whereabouts.d/whereabouts.conf
2023-01-03T09:32:19Z [verbose] pool range [10.250.200.0/24]
2023-01-03T09:32:19Z [verbose] result of garbage collecting pods: <nil>
2023-01-03T09:36:05Z [verbose] deleted pod [default/virt-launcher-vlan200-vm1-4v5jl]
2023-01-03T09:36:05Z [verbose] skipped net-attach-def for default network
2023-01-03T09:36:05Z [debug] pod's network status: {Name:default/br-vlan-200 Interface:net1 IPs:[10.250.200.10] Mac:4a:9a:2f:9d:81:60 Default:false DNS:{Nameservers:[] Domain: Search:[] Options:[]} DeviceInfo:<nil>}
2023-01-03T09:36:05Z [verbose] the NAD's config: {{ "cniVersion": "0.3.1", "type": "bridge", "bridge": "br0", "vlan": 200, "ipam": { "type": "whereabouts", "range": "10.250.200.10-10.250.200.250/24", "gateway": "10.250.200.1", "log_level": "debug", "dns": { "nameservers":[ "8.8.8.8", "1.1.1.1" ] }, "routes": [ { "dst": "0.0.0.0/0", "gw": "10.250.200.1" }, { "dst": "10.250.0.0/24", "gw": "10.250.200.1" } ] } }}
2023-01-03T09:36:05Z [debug] Used defaults from parsed flat file config @ /host/etc/cni/net.d/whereabouts.d/whereabouts.conf
2023-01-03T09:36:05Z [verbose] pool range [10.250.200.0/24]
2023-01-03T09:36:05Z [verbose] result of garbage collecting pods: <nil>