No connectivity between some of the bridged memif interfaces
xzfc opened this issue · 11 comments
xzfc commented
Steps to reproduce
Files to reproduce the problem: bridge-test.tar.gz.
-
Run
etcdand two vpp-agent containers with a shared volume:./run.sh etcd,./run.sh compose. -
Apply the following configuration using
./run.sh config:
- The
bridgecontainer bridges 40 master memif sockets placed in a shared directory. - The
forwardercontainer converts these memifs into kernel interfaces trough xconnect pairs.
image
vppagent-bridge.yml
linuxConfig: {}
netallocConfig: {}
vppConfig:
interfaces:
- name: mb-01
type: MEMIF
enabled: true
memif:
master: true
socketFilename: /shared/memif-01.sock
- name: mb-02
type: MEMIF
enabled: true
memif:
master: true
socketFilename: /shared/memif-02.sock
- name: mb-03
type: MEMIF
enabled: true
memif:
master: true
socketFilename: /shared/memif-03.sock
- name: mb-04
type: MEMIF
enabled: true
memif:
master: true
socketFilename: /shared/memif-04.sock
- name: mb-05
type: MEMIF
enabled: true
memif:
master: true
socketFilename: /shared/memif-05.sock
- name: mb-06
type: MEMIF
enabled: true
memif:
master: true
socketFilename: /shared/memif-06.sock
- name: mb-07
type: MEMIF
enabled: true
memif:
master: true
socketFilename: /shared/memif-07.sock
- name: mb-08
type: MEMIF
enabled: true
memif:
master: true
socketFilename: /shared/memif-08.sock
- name: mb-09
type: MEMIF
enabled: true
memif:
master: true
socketFilename: /shared/memif-09.sock
- name: mb-10
type: MEMIF
enabled: true
memif:
master: true
socketFilename: /shared/memif-10.sock
- name: mb-11
type: MEMIF
enabled: true
memif:
master: true
socketFilename: /shared/memif-11.sock
- name: mb-12
type: MEMIF
enabled: true
memif:
master: true
socketFilename: /shared/memif-12.sock
- name: mb-13
type: MEMIF
enabled: true
memif:
master: true
socketFilename: /shared/memif-13.sock
- name: mb-14
type: MEMIF
enabled: true
memif:
master: true
socketFilename: /shared/memif-14.sock
- name: mb-15
type: MEMIF
enabled: true
memif:
master: true
socketFilename: /shared/memif-15.sock
- name: mb-16
type: MEMIF
enabled: true
memif:
master: true
socketFilename: /shared/memif-16.sock
- name: mb-17
type: MEMIF
enabled: true
memif:
master: true
socketFilename: /shared/memif-17.sock
- name: mb-18
type: MEMIF
enabled: true
memif:
master: true
socketFilename: /shared/memif-18.sock
- name: mb-19
type: MEMIF
enabled: true
memif:
master: true
socketFilename: /shared/memif-19.sock
- name: mb-20
type: MEMIF
enabled: true
memif:
master: true
socketFilename: /shared/memif-20.sock
- name: mb-21
type: MEMIF
enabled: true
memif:
master: true
socketFilename: /shared/memif-21.sock
- name: mb-22
type: MEMIF
enabled: true
memif:
master: true
socketFilename: /shared/memif-22.sock
- name: mb-23
type: MEMIF
enabled: true
memif:
master: true
socketFilename: /shared/memif-23.sock
- name: mb-24
type: MEMIF
enabled: true
memif:
master: true
socketFilename: /shared/memif-24.sock
- name: mb-25
type: MEMIF
enabled: true
memif:
master: true
socketFilename: /shared/memif-25.sock
- name: mb-26
type: MEMIF
enabled: true
memif:
master: true
socketFilename: /shared/memif-26.sock
- name: mb-27
type: MEMIF
enabled: true
memif:
master: true
socketFilename: /shared/memif-27.sock
- name: mb-28
type: MEMIF
enabled: true
memif:
master: true
socketFilename: /shared/memif-28.sock
- name: mb-29
type: MEMIF
enabled: true
memif:
master: true
socketFilename: /shared/memif-29.sock
- name: mb-30
type: MEMIF
enabled: true
memif:
master: true
socketFilename: /shared/memif-30.sock
- name: mb-31
type: MEMIF
enabled: true
memif:
master: true
socketFilename: /shared/memif-31.sock
- name: mb-32
type: MEMIF
enabled: true
memif:
master: true
socketFilename: /shared/memif-32.sock
- name: mb-33
type: MEMIF
enabled: true
memif:
master: true
socketFilename: /shared/memif-33.sock
- name: mb-34
type: MEMIF
enabled: true
memif:
master: true
socketFilename: /shared/memif-34.sock
- name: mb-35
type: MEMIF
enabled: true
memif:
master: true
socketFilename: /shared/memif-35.sock
- name: mb-36
type: MEMIF
enabled: true
memif:
master: true
socketFilename: /shared/memif-36.sock
- name: mb-37
type: MEMIF
enabled: true
memif:
master: true
socketFilename: /shared/memif-37.sock
- name: mb-38
type: MEMIF
enabled: true
memif:
master: true
socketFilename: /shared/memif-38.sock
- name: mb-39
type: MEMIF
enabled: true
memif:
master: true
socketFilename: /shared/memif-39.sock
- name: mb-40
type: MEMIF
enabled: true
memif:
master: true
socketFilename: /shared/memif-40.sock
bridgeDomains:
- name: brd
flood: true
unknownUnicastFlood: true
forward: true
learn: true
macAge: 120
interfaces:
- name: mb-01
- name: mb-02
- name: mb-03
- name: mb-04
- name: mb-05
- name: mb-06
- name: mb-07
- name: mb-08
- name: mb-09
- name: mb-10
- name: mb-11
- name: mb-12
- name: mb-13
- name: mb-14
- name: mb-15
- name: mb-16
- name: mb-17
- name: mb-18
- name: mb-19
- name: mb-20
- name: mb-21
- name: mb-22
- name: mb-23
- name: mb-24
- name: mb-25
- name: mb-26
- name: mb-27
- name: mb-28
- name: mb-29
- name: mb-30
- name: mb-31
- name: mb-32
- name: mb-33
- name: mb-34
- name: mb-35
- name: mb-36
- name: mb-37
- name: mb-38
- name: mb-39
- name: mb-40
vppagent-forwarder.yml
vppConfig:
interfaces:
- name: mf-01
type: MEMIF
enabled: true
memif:
socketFilename: /shared/memif-01.sock
- name: tap-01
type: TAP
enabled: true
tap:
version: 2
- name: mf-02
type: MEMIF
enabled: true
memif:
socketFilename: /shared/memif-02.sock
- name: tap-02
type: TAP
enabled: true
tap:
version: 2
- name: mf-03
type: MEMIF
enabled: true
memif:
socketFilename: /shared/memif-03.sock
- name: tap-03
type: TAP
enabled: true
tap:
version: 2
- name: mf-04
type: MEMIF
enabled: true
memif:
socketFilename: /shared/memif-04.sock
- name: tap-04
type: TAP
enabled: true
tap:
version: 2
- name: mf-05
type: MEMIF
enabled: true
memif:
socketFilename: /shared/memif-05.sock
- name: tap-05
type: TAP
enabled: true
tap:
version: 2
- name: mf-06
type: MEMIF
enabled: true
memif:
socketFilename: /shared/memif-06.sock
- name: tap-06
type: TAP
enabled: true
tap:
version: 2
- name: mf-07
type: MEMIF
enabled: true
memif:
socketFilename: /shared/memif-07.sock
- name: tap-07
type: TAP
enabled: true
tap:
version: 2
- name: mf-08
type: MEMIF
enabled: true
memif:
socketFilename: /shared/memif-08.sock
- name: tap-08
type: TAP
enabled: true
tap:
version: 2
- name: mf-09
type: MEMIF
enabled: true
memif:
socketFilename: /shared/memif-09.sock
- name: tap-09
type: TAP
enabled: true
tap:
version: 2
- name: mf-10
type: MEMIF
enabled: true
memif:
socketFilename: /shared/memif-10.sock
- name: tap-10
type: TAP
enabled: true
tap:
version: 2
- name: mf-11
type: MEMIF
enabled: true
memif:
socketFilename: /shared/memif-11.sock
- name: tap-11
type: TAP
enabled: true
tap:
version: 2
- name: mf-12
type: MEMIF
enabled: true
memif:
socketFilename: /shared/memif-12.sock
- name: tap-12
type: TAP
enabled: true
tap:
version: 2
- name: mf-13
type: MEMIF
enabled: true
memif:
socketFilename: /shared/memif-13.sock
- name: tap-13
type: TAP
enabled: true
tap:
version: 2
- name: mf-14
type: MEMIF
enabled: true
memif:
socketFilename: /shared/memif-14.sock
- name: tap-14
type: TAP
enabled: true
tap:
version: 2
- name: mf-15
type: MEMIF
enabled: true
memif:
socketFilename: /shared/memif-15.sock
- name: tap-15
type: TAP
enabled: true
tap:
version: 2
- name: mf-16
type: MEMIF
enabled: true
memif:
socketFilename: /shared/memif-16.sock
- name: tap-16
type: TAP
enabled: true
tap:
version: 2
- name: mf-17
type: MEMIF
enabled: true
memif:
socketFilename: /shared/memif-17.sock
- name: tap-17
type: TAP
enabled: true
tap:
version: 2
- name: mf-18
type: MEMIF
enabled: true
memif:
socketFilename: /shared/memif-18.sock
- name: tap-18
type: TAP
enabled: true
tap:
version: 2
- name: mf-19
type: MEMIF
enabled: true
memif:
socketFilename: /shared/memif-19.sock
- name: tap-19
type: TAP
enabled: true
tap:
version: 2
- name: mf-20
type: MEMIF
enabled: true
memif:
socketFilename: /shared/memif-20.sock
- name: tap-20
type: TAP
enabled: true
tap:
version: 2
- name: mf-21
type: MEMIF
enabled: true
memif:
socketFilename: /shared/memif-21.sock
- name: tap-21
type: TAP
enabled: true
tap:
version: 2
- name: mf-22
type: MEMIF
enabled: true
memif:
socketFilename: /shared/memif-22.sock
- name: tap-22
type: TAP
enabled: true
tap:
version: 2
- name: mf-23
type: MEMIF
enabled: true
memif:
socketFilename: /shared/memif-23.sock
- name: tap-23
type: TAP
enabled: true
tap:
version: 2
- name: mf-24
type: MEMIF
enabled: true
memif:
socketFilename: /shared/memif-24.sock
- name: tap-24
type: TAP
enabled: true
tap:
version: 2
- name: mf-25
type: MEMIF
enabled: true
memif:
socketFilename: /shared/memif-25.sock
- name: tap-25
type: TAP
enabled: true
tap:
version: 2
- name: mf-26
type: MEMIF
enabled: true
memif:
socketFilename: /shared/memif-26.sock
- name: tap-26
type: TAP
enabled: true
tap:
version: 2
- name: mf-27
type: MEMIF
enabled: true
memif:
socketFilename: /shared/memif-27.sock
- name: tap-27
type: TAP
enabled: true
tap:
version: 2
- name: mf-28
type: MEMIF
enabled: true
memif:
socketFilename: /shared/memif-28.sock
- name: tap-28
type: TAP
enabled: true
tap:
version: 2
- name: mf-29
type: MEMIF
enabled: true
memif:
socketFilename: /shared/memif-29.sock
- name: tap-29
type: TAP
enabled: true
tap:
version: 2
- name: mf-30
type: MEMIF
enabled: true
memif:
socketFilename: /shared/memif-30.sock
- name: tap-30
type: TAP
enabled: true
tap:
version: 2
- name: mf-31
type: MEMIF
enabled: true
memif:
socketFilename: /shared/memif-31.sock
- name: tap-31
type: TAP
enabled: true
tap:
version: 2
- name: mf-32
type: MEMIF
enabled: true
memif:
socketFilename: /shared/memif-32.sock
- name: tap-32
type: TAP
enabled: true
tap:
version: 2
- name: mf-33
type: MEMIF
enabled: true
memif:
socketFilename: /shared/memif-33.sock
- name: tap-33
type: TAP
enabled: true
tap:
version: 2
- name: mf-34
type: MEMIF
enabled: true
memif:
socketFilename: /shared/memif-34.sock
- name: tap-34
type: TAP
enabled: true
tap:
version: 2
- name: mf-35
type: MEMIF
enabled: true
memif:
socketFilename: /shared/memif-35.sock
- name: tap-35
type: TAP
enabled: true
tap:
version: 2
- name: mf-36
type: MEMIF
enabled: true
memif:
socketFilename: /shared/memif-36.sock
- name: tap-36
type: TAP
enabled: true
tap:
version: 2
- name: mf-37
type: MEMIF
enabled: true
memif:
socketFilename: /shared/memif-37.sock
- name: tap-37
type: TAP
enabled: true
tap:
version: 2
- name: mf-38
type: MEMIF
enabled: true
memif:
socketFilename: /shared/memif-38.sock
- name: tap-38
type: TAP
enabled: true
tap:
version: 2
- name: mf-39
type: MEMIF
enabled: true
memif:
socketFilename: /shared/memif-39.sock
- name: tap-39
type: TAP
enabled: true
tap:
version: 2
- name: mf-40
type: MEMIF
enabled: true
memif:
socketFilename: /shared/memif-40.sock
- name: tap-40
type: TAP
enabled: true
tap:
version: 2
xconnectPairs:
- receiveInterface: mf-01
transmitInterface: tap-01
- receiveInterface: tap-01
transmitInterface: mf-01
- receiveInterface: mf-02
transmitInterface: tap-02
- receiveInterface: tap-02
transmitInterface: mf-02
- receiveInterface: mf-03
transmitInterface: tap-03
- receiveInterface: tap-03
transmitInterface: mf-03
- receiveInterface: mf-04
transmitInterface: tap-04
- receiveInterface: tap-04
transmitInterface: mf-04
- receiveInterface: mf-05
transmitInterface: tap-05
- receiveInterface: tap-05
transmitInterface: mf-05
- receiveInterface: mf-06
transmitInterface: tap-06
- receiveInterface: tap-06
transmitInterface: mf-06
- receiveInterface: mf-07
transmitInterface: tap-07
- receiveInterface: tap-07
transmitInterface: mf-07
- receiveInterface: mf-08
transmitInterface: tap-08
- receiveInterface: tap-08
transmitInterface: mf-08
- receiveInterface: mf-09
transmitInterface: tap-09
- receiveInterface: tap-09
transmitInterface: mf-09
- receiveInterface: mf-10
transmitInterface: tap-10
- receiveInterface: tap-10
transmitInterface: mf-10
- receiveInterface: mf-11
transmitInterface: tap-11
- receiveInterface: tap-11
transmitInterface: mf-11
- receiveInterface: mf-12
transmitInterface: tap-12
- receiveInterface: tap-12
transmitInterface: mf-12
- receiveInterface: mf-13
transmitInterface: tap-13
- receiveInterface: tap-13
transmitInterface: mf-13
- receiveInterface: mf-14
transmitInterface: tap-14
- receiveInterface: tap-14
transmitInterface: mf-14
- receiveInterface: mf-15
transmitInterface: tap-15
- receiveInterface: tap-15
transmitInterface: mf-15
- receiveInterface: mf-16
transmitInterface: tap-16
- receiveInterface: tap-16
transmitInterface: mf-16
- receiveInterface: mf-17
transmitInterface: tap-17
- receiveInterface: tap-17
transmitInterface: mf-17
- receiveInterface: mf-18
transmitInterface: tap-18
- receiveInterface: tap-18
transmitInterface: mf-18
- receiveInterface: mf-19
transmitInterface: tap-19
- receiveInterface: tap-19
transmitInterface: mf-19
- receiveInterface: mf-20
transmitInterface: tap-20
- receiveInterface: tap-20
transmitInterface: mf-20
- receiveInterface: mf-21
transmitInterface: tap-21
- receiveInterface: tap-21
transmitInterface: mf-21
- receiveInterface: mf-22
transmitInterface: tap-22
- receiveInterface: tap-22
transmitInterface: mf-22
- receiveInterface: mf-23
transmitInterface: tap-23
- receiveInterface: tap-23
transmitInterface: mf-23
- receiveInterface: mf-24
transmitInterface: tap-24
- receiveInterface: tap-24
transmitInterface: mf-24
- receiveInterface: mf-25
transmitInterface: tap-25
- receiveInterface: tap-25
transmitInterface: mf-25
- receiveInterface: mf-26
transmitInterface: tap-26
- receiveInterface: tap-26
transmitInterface: mf-26
- receiveInterface: mf-27
transmitInterface: tap-27
- receiveInterface: tap-27
transmitInterface: mf-27
- receiveInterface: mf-28
transmitInterface: tap-28
- receiveInterface: tap-28
transmitInterface: mf-28
- receiveInterface: mf-29
transmitInterface: tap-29
- receiveInterface: tap-29
transmitInterface: mf-29
- receiveInterface: mf-30
transmitInterface: tap-30
- receiveInterface: tap-30
transmitInterface: mf-30
- receiveInterface: mf-31
transmitInterface: tap-31
- receiveInterface: tap-31
transmitInterface: mf-31
- receiveInterface: mf-32
transmitInterface: tap-32
- receiveInterface: tap-32
transmitInterface: mf-32
- receiveInterface: mf-33
transmitInterface: tap-33
- receiveInterface: tap-33
transmitInterface: mf-33
- receiveInterface: mf-34
transmitInterface: tap-34
- receiveInterface: tap-34
transmitInterface: mf-34
- receiveInterface: mf-35
transmitInterface: tap-35
- receiveInterface: tap-35
transmitInterface: mf-35
- receiveInterface: mf-36
transmitInterface: tap-36
- receiveInterface: tap-36
transmitInterface: mf-36
- receiveInterface: mf-37
transmitInterface: tap-37
- receiveInterface: tap-37
transmitInterface: mf-37
- receiveInterface: mf-38
transmitInterface: tap-38
- receiveInterface: tap-38
transmitInterface: mf-38
- receiveInterface: mf-39
transmitInterface: tap-39
- receiveInterface: tap-39
transmitInterface: mf-39
- receiveInterface: mf-40
transmitInterface: tap-40
- receiveInterface: tap-40
transmitInterface: mf-40
linuxConfig:
interfaces:
- name: i-01
type: TAP_TO_VPP
hostIfName: i-01
enabled: true
ipAddresses:
- 172.16.0.1/24
tap:
vppTapIfName: tap-01
- name: i-02
type: TAP_TO_VPP
hostIfName: i-02
enabled: true
ipAddresses:
- 172.16.0.2/24
tap:
vppTapIfName: tap-02
- name: i-03
type: TAP_TO_VPP
hostIfName: i-03
enabled: true
ipAddresses:
- 172.16.0.3/24
tap:
vppTapIfName: tap-03
- name: i-04
type: TAP_TO_VPP
hostIfName: i-04
enabled: true
ipAddresses:
- 172.16.0.4/24
tap:
vppTapIfName: tap-04
- name: i-05
type: TAP_TO_VPP
hostIfName: i-05
enabled: true
ipAddresses:
- 172.16.0.5/24
tap:
vppTapIfName: tap-05
- name: i-06
type: TAP_TO_VPP
hostIfName: i-06
enabled: true
ipAddresses:
- 172.16.0.6/24
tap:
vppTapIfName: tap-06
- name: i-07
type: TAP_TO_VPP
hostIfName: i-07
enabled: true
ipAddresses:
- 172.16.0.7/24
tap:
vppTapIfName: tap-07
- name: i-08
type: TAP_TO_VPP
hostIfName: i-08
enabled: true
ipAddresses:
- 172.16.0.8/24
tap:
vppTapIfName: tap-08
- name: i-09
type: TAP_TO_VPP
hostIfName: i-09
enabled: true
ipAddresses:
- 172.16.0.9/24
tap:
vppTapIfName: tap-09
- name: i-10
type: TAP_TO_VPP
hostIfName: i-10
enabled: true
ipAddresses:
- 172.16.0.10/24
tap:
vppTapIfName: tap-10
- name: i-11
type: TAP_TO_VPP
hostIfName: i-11
enabled: true
ipAddresses:
- 172.16.0.11/24
tap:
vppTapIfName: tap-11
- name: i-12
type: TAP_TO_VPP
hostIfName: i-12
enabled: true
ipAddresses:
- 172.16.0.12/24
tap:
vppTapIfName: tap-12
- name: i-13
type: TAP_TO_VPP
hostIfName: i-13
enabled: true
ipAddresses:
- 172.16.0.13/24
tap:
vppTapIfName: tap-13
- name: i-14
type: TAP_TO_VPP
hostIfName: i-14
enabled: true
ipAddresses:
- 172.16.0.14/24
tap:
vppTapIfName: tap-14
- name: i-15
type: TAP_TO_VPP
hostIfName: i-15
enabled: true
ipAddresses:
- 172.16.0.15/24
tap:
vppTapIfName: tap-15
- name: i-16
type: TAP_TO_VPP
hostIfName: i-16
enabled: true
ipAddresses:
- 172.16.0.16/24
tap:
vppTapIfName: tap-16
- name: i-17
type: TAP_TO_VPP
hostIfName: i-17
enabled: true
ipAddresses:
- 172.16.0.17/24
tap:
vppTapIfName: tap-17
- name: i-18
type: TAP_TO_VPP
hostIfName: i-18
enabled: true
ipAddresses:
- 172.16.0.18/24
tap:
vppTapIfName: tap-18
- name: i-19
type: TAP_TO_VPP
hostIfName: i-19
enabled: true
ipAddresses:
- 172.16.0.19/24
tap:
vppTapIfName: tap-19
- name: i-20
type: TAP_TO_VPP
hostIfName: i-20
enabled: true
ipAddresses:
- 172.16.0.20/24
tap:
vppTapIfName: tap-20
- name: i-21
type: TAP_TO_VPP
hostIfName: i-21
enabled: true
ipAddresses:
- 172.16.0.21/24
tap:
vppTapIfName: tap-21
- name: i-22
type: TAP_TO_VPP
hostIfName: i-22
enabled: true
ipAddresses:
- 172.16.0.22/24
tap:
vppTapIfName: tap-22
- name: i-23
type: TAP_TO_VPP
hostIfName: i-23
enabled: true
ipAddresses:
- 172.16.0.23/24
tap:
vppTapIfName: tap-23
- name: i-24
type: TAP_TO_VPP
hostIfName: i-24
enabled: true
ipAddresses:
- 172.16.0.24/24
tap:
vppTapIfName: tap-24
- name: i-25
type: TAP_TO_VPP
hostIfName: i-25
enabled: true
ipAddresses:
- 172.16.0.25/24
tap:
vppTapIfName: tap-25
- name: i-26
type: TAP_TO_VPP
hostIfName: i-26
enabled: true
ipAddresses:
- 172.16.0.26/24
tap:
vppTapIfName: tap-26
- name: i-27
type: TAP_TO_VPP
hostIfName: i-27
enabled: true
ipAddresses:
- 172.16.0.27/24
tap:
vppTapIfName: tap-27
- name: i-28
type: TAP_TO_VPP
hostIfName: i-28
enabled: true
ipAddresses:
- 172.16.0.28/24
tap:
vppTapIfName: tap-28
- name: i-29
type: TAP_TO_VPP
hostIfName: i-29
enabled: true
ipAddresses:
- 172.16.0.29/24
tap:
vppTapIfName: tap-29
- name: i-30
type: TAP_TO_VPP
hostIfName: i-30
enabled: true
ipAddresses:
- 172.16.0.30/24
tap:
vppTapIfName: tap-30
- name: i-31
type: TAP_TO_VPP
hostIfName: i-31
enabled: true
ipAddresses:
- 172.16.0.31/24
tap:
vppTapIfName: tap-31
- name: i-32
type: TAP_TO_VPP
hostIfName: i-32
enabled: true
ipAddresses:
- 172.16.0.32/24
tap:
vppTapIfName: tap-32
- name: i-33
type: TAP_TO_VPP
hostIfName: i-33
enabled: true
ipAddresses:
- 172.16.0.33/24
tap:
vppTapIfName: tap-33
- name: i-34
type: TAP_TO_VPP
hostIfName: i-34
enabled: true
ipAddresses:
- 172.16.0.34/24
tap:
vppTapIfName: tap-34
- name: i-35
type: TAP_TO_VPP
hostIfName: i-35
enabled: true
ipAddresses:
- 172.16.0.35/24
tap:
vppTapIfName: tap-35
- name: i-36
type: TAP_TO_VPP
hostIfName: i-36
enabled: true
ipAddresses:
- 172.16.0.36/24
tap:
vppTapIfName: tap-36
- name: i-37
type: TAP_TO_VPP
hostIfName: i-37
enabled: true
ipAddresses:
- 172.16.0.37/24
tap:
vppTapIfName: tap-37
- name: i-38
type: TAP_TO_VPP
hostIfName: i-38
enabled: true
ipAddresses:
- 172.16.0.38/24
tap:
vppTapIfName: tap-38
- name: i-39
type: TAP_TO_VPP
hostIfName: i-39
enabled: true
ipAddresses:
- 172.16.0.39/24
tap:
vppTapIfName: tap-39
- name: i-40
type: TAP_TO_VPP
hostIfName: i-40
enabled: true
ipAddresses:
- 172.16.0.40/24
tap:
vppTapIfName: tap-40
netallocConfig: {}
-
Watch interface rx/tx counters inside the
forwardercontainer using./run.sh stats(or alternativelyip -s link). -
Broadcast some frames from interface
i-01: use./run.sh pingto send 100 pings.
Expected result
Sent pings should be broadcasted on all interfaces, i.e. rx counter on i-02..i-40 should be equal to 100.
Actual result
Pings are received only on some of the first interfaces.
./run.sh stats output
rx tx
eth0: 578 618
i-01: 0 100
i-02: 100 0
i-03: 100 0
i-04: 100 0
i-05: 100 0
i-06: 100 0
i-07: 100 0
i-08: 0 0
i-09: 0 0
i-10: 0 0
i-11: 0 0
i-12: 0 0
i-13: 0 0
i-14: 0 0
i-15: 0 0
i-16: 0 0
i-17: 0 0
i-18: 0 0
i-19: 0 0
i-20: 0 0
i-21: 0 0
i-22: 0 0
i-23: 0 0
i-24: 0 0
i-25: 0 0
i-26: 0 0
i-27: 0 0
i-28: 0 0
i-29: 0 0
i-30: 0 0
i-31: 0 0
i-32: 0 0
i-33: 0 0
i-34: 0 0
i-35: 0 0
i-36: 0 0
i-37: 0 0
i-38: 0 0
i-39: 0 0
i-40: 0 0
lo: 38 38
Vppagent output with INITIAL_LOGLVL=debug enabled: vpp-agent-log.txt.
Version
__
_ _____ ___ _______ ____ ____ ___ / /_
| |/ / _ \/ _ /___/ _ '/ _ '/ -_/ _ / __/ vpp-agent v3.2.0-alpha-76-g680315bf0
|___/ .__/ .__/ \_'_/\_' /\__/_//_\__/ Mon Oct 26 16:09:46 UTC 2020 (8 days ago)
/_/ /_/ /___/ root@5dc2fef99957 (go1.15.2 linux/amd64)
vpp# show version
vpp v20.05.1-6~gf53edbc3b~b1 built by root on c6de781be75d at 2020-10-12T15:25:28
This issue is a supposed cause of the NSM issue networkservicemesh/networkservicemesh#2208. Not sure whether it is a vpp or vpp-agent issue.
ondrej-fabry commented
This seems like issue with the actual VPP configuration for this setup.
Running vppctl show err on VPP in bridge container shows this:
vpp# sh err
Count Node Reason
1560 l2-output L2 output packets
40 l2-learn L2 learn packets
1 l2-learn L2 learn misses
3 l2-learn L2 learn hit updates
40 l2-input L2 input packets
40 l2-flood L2 flood packets
40 memif8/0-tx no free tx slots
40 memif9/0-tx no free tx slots
40 memif10/0-tx no free tx slots
40 memif11/0-tx no free tx slots
40 memif12/0-tx no free tx slots
40 memif13/0-tx no free tx slots
40 memif14/0-tx no free tx slots
40 memif15/0-tx no free tx slots
40 memif16/0-tx no free tx slots
40 memif17/0-tx no free tx slots
40 memif18/0-tx no free tx slots
40 memif19/0-tx no free tx slots
40 memif20/0-tx no free tx slots
40 memif21/0-tx no free tx slots
40 memif22/0-tx no free tx slots
40 memif23/0-tx no free tx slots
40 memif24/0-tx no free tx slots
40 memif25/0-tx no free tx slots
40 memif26/0-tx no free tx slots
40 memif27/0-tx no free tx slots
40 memif28/0-tx no free tx slots
40 memif29/0-tx no free tx slots
40 memif30/0-tx no free tx slots
40 memif31/0-tx no free tx slots
40 memif32/0-tx no free tx slots
40 memif33/0-tx no free tx slots
40 memif34/0-tx no free tx slots
40 memif35/0-tx no free tx slots
40 memif36/0-tx no free tx slots
40 memif37/0-tx no free tx slots
40 memif38/0-tx no free tx slots
40 memif39/0-tx no free tx slots
40 memif40/0-tx no free tx slots
Seems like some config needs to be tweaked for memif interfaces or bridge domain, not sure.
On VPP in forwarder container:
vpp# sh err
Count Node Reason
22576651 memif-input buffer allocation failed
280 l2-output L2 output packets
280 l2-input L2 input packets
Here is packet trace from VPP in bridge container (vppctl trace add memif-input 10): https://pastebin.com/raw/NHvDGdw1
Also, it would be helpful if you could provide vpp-agent logs with similar issues - ideally with INITIAL_LOGLVL=debug env var set.
xzfc commented
Similar issue with a single container, taps, and xconnects (no bridge and no memifs).
image
Code to reproduce: bridge-test2.tar.gz.
Steps are the same as in the first message. ./run.sh ping sends 100 pings from each interface.
Expected result: all interface counters should be equal to 100. Actual result: some counters are 0.
./run.sh stats output
rx tx
a-01: 100 100
a-02: 100 100
a-03: 100 100
a-04: 100 100
a-05: 100 100
a-06: 100 100
a-07: 100 100
a-08: 100 100
a-09: 100 100
a-10: 100 100
a-11: 100 100
a-12: 100 100
a-13: 100 100
a-14: 100 100
a-15: 100 100
a-16: 100 100
a-17: 100 100
a-18: 100 100
a-19: 100 100
a-20: 100 100
a-21: 100 100
a-22: 100 100
a-23: 100 100
a-24: 100 100
a-25: 100 100
a-26: 100 100
a-27: 100 100
a-28: 100 100
a-29: 100 100
a-30: 100 100
a-31: 100 100
a-32: 100 100
a-33: 100 100
a-34: 100 100
a-35: 100 100
a-36: 100 100
a-37: 0 100
a-38: 0 100
a-39: 0 100
a-40: 0 100
b-01: 100 100
b-02: 100 100
b-03: 100 100
b-04: 100 100
b-05: 100 100
b-06: 100 100
b-07: 100 100
b-08: 100 100
b-09: 100 100
b-10: 100 100
b-11: 100 100
b-12: 100 100
b-13: 100 100
b-14: 100 100
b-15: 100 100
b-16: 100 100
b-17: 100 100
b-18: 100 100
b-19: 100 100
b-20: 100 100
b-21: 100 100
b-22: 100 100
b-23: 100 100
b-24: 100 100
b-25: 100 100
b-26: 100 100
b-27: 100 100
b-28: 100 100
b-29: 100 100
b-30: 100 100
b-31: 100 100
b-32: 100 100
b-33: 100 100
b-34: 100 100
b-35: 100 100
b-36: 100 100
b-37: 100 0
b-38: 100 0
b-39: 100 0
b-40: 100 0
eth0: 988 1072
lo: 40 40
Applied config: vppagent.yml.txt.
Vppagent output with INITIAL_LOGLVL=debug enabled: vpp-agent-log.txt.
vpp# show err
Count Node Reason
7600 l2-output L2 output packets
7600 l2-input L2 input packets
milanlenco commented
Are all these TAPs in the same namespace?
interfaces:
- name: i-01
type: TAP_TO_VPP
hostIfName: i-01
enabled: true
ipAddresses:
- 172.16.0.1/24
tap:
vppTapIfName: tap-01
- name: i-02
type: TAP_TO_VPP
hostIfName: i-02
enabled: true
ipAddresses:
- 172.16.0.2/24
tap:
vppTapIfName: tap-02
- name: i-03
type: TAP_TO_VPP
hostIfName: i-03
enabled: true
ipAddresses:
- 172.16.0.3/24
tap:
vppTapIfName: tap-03
If so than you might be running into the "ARP Flux" issue on the Linux.
xzfc commented
Are all these TAPs in the same namespace?
If so than you might be running into the "ARP Flux" issue on the Linux.
Checked it with raw ethernet frames instead of pings (so, no IP/ARP involved), and put each interface pair in a separate subnet. The issue persists. (updated code: bridge-test2-raw.tar.gz, ./run.sh broadcast-raw)
Also, it would be helpful if you could provide vpp-agent logs with similar issues - ideally with
INITIAL_LOGLVL=debugenv var set.
Updated the issue description to include this log.
Could you please clarify the status of this issue: can we (NSM devs) expect that you'll take a care of this, or should we report it to VPP on our own (if it is a VPP issue)?
milanlenco commented
Seems more like a VPP issue. But to really be sure you could try to replicate the scenario using VPP CLIs (i.e. without the agent).
xzfc commented
The issue is solved for both of the described scenarios by tuning buffers-per-numa in the vpp config.
I'm closing this issue as it is not related to vpp-agent. Thanks for your time.
buffers {
buffers-per-numa 64000 # Default is 16384.
# 64000 is enough to handle 40 connections as in the scenario described in the issue description.
}
xzfc commented
Would it make sense for vpp-agent to keep track of used buffers and return an error on creating an interface if there are no enough free buffers?
ondrej-fabry commented
That sounds kind of useful for sure.. but I think there is no deterministic way to know for sure how much buffer space interfaces use - various interface types might use different amount and also it is something that might change between versions and would require keeping track.
Perhaps, a better solution for such cases would be to check errors/counters for interfaces and propagate it through vpp-agent in some way. This would actually solve many other types of issues related to any problems with interfaces.
@milanlenco thoughts?
milanlenco commented
I would expect VPP to return error if you try to create interface and there are not enough resources for it. Such error would be naturally propagated through the agent. If VPP just silently ignores the lack of resources then it seems like a bug to me and perhaps you could raise this issue with VPP developers (?).
xzfc commented
Checked create tap in vppctl. It just silently creates an interface regardless of free buffers. However, show tap reports a different desc_in_use value based on whether the initial allocation succeeded.
…
Virtqueue (RX) 0
qsz 256, last_used_idx 0, desc_next 0, desc_in_use 256 # ok
…
Virtqueue (RX) 0
qsz 256, last_used_idx 0, desc_next 0, desc_in_use 0 # bad
…
Probably, different interface types have a similar check.
ondrej-fabry commented
I would expect VPP to return error if you try to create interface and there are not enough resources for it. Such error would be naturally propagated through the agent. If VPP just silently ignores the lack of resources then it seems like a bug to me and perhaps you could raise this issue with VPP developers (?).
Totally agree with Milan here. Not sure why it allows creating interfaces for which it does not have any space.. maybe it allows cases in which some interfaces can start working after space is freed (other interface is deleted) or something like that.. Best to ask some VPP developers about this to be sure.
Checked
create tapinvppctl. It just silently creates an interface regardless of free buffers. However,show tapreports a differentdesc_in_usevalue based on whether the initial allocation succeeded.… Virtqueue (RX) 0 qsz 256, last_used_idx 0, desc_next 0, desc_in_use 256 # ok … Virtqueue (RX) 0 qsz 256, last_used_idx 0, desc_next 0, desc_in_use 0 # bad …Probably, different interface types have a similar check.
@xzfc Understandably, for a certain VPP version, specific type of interface and by using correct CLI commands you might be able to retrieve various information about individual interfaces and then use the output from that command to check particular values to determine if the interface is OK or not. However using process like this is not really intended to be used by control-planes as it is absolutely unmaintainable and would require hard-coding a lot of stuff to make it work for all types of interfaces, unaffected by changes in formatting of CLI output or expected values, ranges etc..