OpenFlow rules missing for service/node after certain period of depoloyment
Opened this issue · 0 comments
Description
I have deployed a customized k8s cluster and on top of that installed Contiv as CNI networking.
All worked fine, but after 2 or 3 days noticed that OpenFlow rules for a specific service/node missing.
Expected Behavior
All the PODs should be accessible from any of the host systems. ie host ---> pod communication
Observed Behavior
Unable to connect to the POD and contivh1 interface IP from other hosts.Due to this unable to connect to the k8s services hosted on a specific host.
After some investigation, I found that the OpenFlow rules for that particular POD IP and contivh1 interface IP are missing.
The is sometime happening for POD IP and sometimes for contivh1 interface IP.
[root@]# kubectl get pod -n kube-system |grep contiv
contiv-netmaster-bj2nc 1/1 Running 0 6d
contiv-netmaster-kqkgq 1/1 Running 0 6d
contiv-netplugin-6rrgc 1/1 Running 0 6d
contiv-netplugin-fdbtd 1/1 Running 0 6d
contiv-netplugin-j7qc9 1/1 Running 0 6d
contiv-netplugin-k4p6q 1/1 Running 0 6d
contiv-netplugin-kpnq9 1/1 Running 0 6d
contiv-netplugin-p6k59 1/1 Running 0 6d
contiv-netplugin-t6zbh 1/1 Running 0 6d
contiv-netplugin-zsd9g 1/1 Running 0 6d
[root@]#
[root@]# kubectl exec contiv-netplugin-6rrgc -n kube-system -- ovs-ofctl -O Openflow13 dump-flows contivVxlanBridge | grep 132.1.1.8
[root@s ~]#
[root@~]# kubectl exec contiv-netplugin-fdbtd -n kube-system -- ovs-ofctl -O Openflow13 dump-flows contivVxlanBridge | grep 132.1.1.8
cookie=0x1dd37, duration=9532.039s, table=4, n_packets=0, n_bytes=0, priority=100,ip,metadata=0x100000000/0xff00000000,nw_dst=132.1.1.8 actions=write_metadata:0/0xfffe,goto_table:5
cookie=0x1dd36, duration=9532.039s, table=7, n_packets=0, n_bytes=0, priority=100,ip,metadata=0x100000000/0xff00000000,nw_dst=132.1.1.8 actions=set_field:0x1->tun_id,set_field:00:00:11:11:11:11->eth_dst,output:267
[root@ ~]#
[root@ ~]# kubectl exec contiv-netplugin-j7qc9 -n kube-system -- ovs-ofctl -O Openflow13 dump-flows contivVxlanBridge | grep 132.1.1.8
cookie=0x499, duration=340671.602s, table=4, n_packets=86, n_bytes=8428, priority=100,ip,metadata=0x100000000/0xff00000000,nw_dst=132.1.1.8 actions=write_metadata:0/0xfffe,goto_table:5
cookie=0x498, duration=340671.603s, table=7, n_packets=86, n_bytes=8428, priority=100,ip,metadata=0x100000000/0xff00000000,nw_dst=132.1.1.8 actions=set_field:0x1->tun_id,set_field:00:00:11:11:11:11->eth_dst,output:12
[root@ ~]#
[root@ ~]# kubectl exec contiv-netplugin-k4p6q -n kube-system -- ovs-ofctl -O Openflow13 dump-flows contivVxlanBridge | grep 132.1.1.8
[root@ ~]#
[root@ ~]# kubectl exec contiv-netplugin-kpnq9 -n kube-system -- ovs-ofctl -O Openflow13 dump-flows contivVxlanBridge | grep 132.1.1.8
cookie=0x1f, duration=604562.586s, table=4, n_packets=329709, n_bytes=31594881, priority=100,ip,metadata=0x100000000/0xff00000000,nw_dst=132.1.1.8 actions=write_metadata:0/0xfffe,goto_table:5
cookie=0x1e, duration=604562.586s, table=7, n_packets=391455, n_bytes=49623858, priority=100,ip,metadata=0x100000000/0xff00000000,nw_dst=132.1.1.8 actions=set_field:00:00:11:11:11:11->eth_src,set_field:02:02:84:01:01:08->eth_dst,output:10
[root ~]#
[root@s~]# kubectl exec contiv-netplugin-p6k59 -n kube-system -- ovs-ofctl -O Openflow13 dump-flows contivVxlanBridge | grep 132.1.1.8
[root@~]#
[root@ ~]# kubectl exec contiv-netplugin-t6zbh -n kube-system -- ovs-ofctl -O Openflow13 dump-flows contivVxlanBridge | grep 132.1.1.8
cookie=0x49f, duration=340798.618s, table=4, n_packets=3, n_bytes=294, priority=100,ip,metadata=0x100000000/0xff00000000,nw_dst=132.1.1.8 actions=write_metadata:0/0xfffe,goto_table:5
cookie=0x49e, duration=340798.618s, table=7, n_packets=3, n_bytes=294, priority=100,ip,metadata=0x100000000/0xff00000000,nw_dst=132.1.1.8 actions=set_field:0x1->tun_id,set_field:00:00:11:11:11:11->eth_dst,output:13
[root@ ~]#
[root@ ~]# kubectl exec contiv-netplugin-zsd9g -n kube-system -- ovs-ofctl -O Openflow13 dump-flows contivVxlanBridge | grep 132.1.1.8
[root@ ~]#