Cannot add MAC address to a VF interface
lcongio opened this issue · 1 comments
lcongio commented
Hi, I have encountered an issue in TRex v3.04 with the Broadcom NetXtreme-E NICs.
If I bind the PFs to DPDK I can run the TRex, but if I bind the VFs I get this error:
bnxt_stats_reset_op(): Device Initialization not complete!
bnxt_mac_addr_add_op(): Cannot add MAC address to a VF interface
Failed setting MAC for port 0
Killing Scapy server... Scapy server is killed
Using TestPMD I can reach the VFs.
The full log with TRex is bellow (the error is at the end):
$ sudo ./t-rex-64 -i -v 7 --cfg /etc/trex_cfg.yaml
Starting Scapy server.... Scapy server is started
The ports are bound/configured.
Starting TRex v3.04 please wait ...
Using configuration file /etc/trex_cfg.yaml
port limit : not configured
port_bandwidth_gb : 10
port_speed : 0
port_mtu : 0
if_mask : None
is low-end : 0
stack type :
thread_per_dual_if : 1
if : 65:02.0, 65:03.0,
enable_zmq_pub : 1
zmq_pub_port : 4500
m_zmq_rpc_port : 4501
src : 00:00:00:00:00:00
dest : 00:00:00:00:00:00
src : 00:00:00:00:00:00
dest : 00:00:00:00:00:00
memory per 2x10G ports
MBUF_64 : 16380
MBUF_128 : 8190
MBUF_256 : 8190
MBUF_512 : 8190
MBUF_1024 : 8190
MBUF_2048 : 4095
MBUF_4096 : 128
MBUF_9K : 512
TRAFFIC_MBUF_64 : 65520
TRAFFIC_MBUF_128 : 32760
TRAFFIC_MBUF_256 : 8190
TRAFFIC_MBUF_512 : 8190
TRAFFIC_MBUF_1024 : 8190
TRAFFIC_MBUF_2048 : 32760
TRAFFIC_MBUF_4096 : 128
TRAFFIC_MBUF_9K : 512
MBUF_DP_FLOWS : 524288
MBUF_GLOBAL_FLOWS : 5120
master thread : 0
rx thread : 15
dual_if : 0
socket : 0
[ 1 2 3 4 5 6 7 8 9 10 11 12 13 14 ]
CTimerWheelYamlInfo does not exist
flags : 8010f00
write_file : 0
verbose : 7
realtime : 1
flip : 0
cores : 1
single core : 0
flow-flip : 0
no clean close : 0
zmq_publish : 1
vlan mode : 0
client_cfg : 0
mbuf_cache_disable : 0
cfg file :
mac file :
out file :
client cfg file :
duration : 0
factor : 1
mbuf_factor : 1
latency : 0 pkt/sec
zmq_port : 4500
telnet_port : 4501
expected_ports : 2
tw_bucket_usec : 20.000000 usec
tw_buckets : 1024 usec
tw_levels : 3 usec
port : 0 dst:00:00:00:00:00:00 src:00:00:00:00:00:00
port : 1 dst:00:00:00:00:00:00 src:00:00:00:00:00:00
port : 2 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 3 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 4 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 5 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 6 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 7 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 8 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 9 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 10 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 11 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 12 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 13 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 14 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 15 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 16 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 17 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 18 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 19 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 20 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 21 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 22 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 23 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 24 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 25 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 26 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 27 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 28 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 29 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 30 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 31 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
Total Memory :
MBUF_64 : 81900
MBUF_128 : 40950
MBUF_256 : 16380
MBUF_512 : 16380
MBUF_1024 : 16380
MBUF_2048 : 36855
MBUF_4096 : 1024
MBUF_DP_FLOWS : 524288
MBUF_GLOBAL_FLOWS : 5120
get_each_core_dp_flows : 524288
Total memory : 248.40 Mbytes
core_list : 0,15,1
sockets : 0
active sockets : 1
ports_sockets : 1
0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
phy | virt
1 1
DPDK args
xx -l 0,15,1 -n 4 --log-level 8 --main-lcore 0 -a 0000:65:02.0 -a 0000:65:03.0 --legacy-mem
EAL: Detected CPU lcores: 16
EAL: Detected NUMA nodes: 1
EAL: Static memory layout is selected, amount of reserved memory can be adjusted with -m or --socket-mem
EAL: Detected static linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'VA'
EAL: VFIO support initialized
EAL: Using IOMMU type 1 (Type 1)
EAL: Probe PCI driver: net_bnxt (14e4:16dc) device: 0000:65:02.0 (socket 0)
EAL: Probe PCI driver: net_bnxt (14e4:16dc) device: 0000:65:03.0 (socket 0)
TELEMETRY: No legacy callbacks, legacy socket not created
input : [65:02.0, 65:03.0]
dpdk : [0000:65:02.0, 0000:65:03.0]
pci_scan : [0000:65:02.0, 0000:65:03.0]
map : [ 0, 1]
TRex port mapping
-----------------
TRex vport: 0 dpdk_rte_eth: 0
TRex vport: 1 dpdk_rte_eth: 1
set driver name net_bnxt
driver capability : TCP_UDP_OFFLOAD TSO LRO
set dpdk queues mode to ONE_QUE
DPDK devices 2 : 2
-----
0 : vdev 0000:65:02.0
1 : vdev 0000:65:03.0
-----
Number of ports found: 2
if_index : 0
driver name : net_bnxt
min_rx_bufsize : 1
max_rx_pktlen : 9600
max_rx_queues : 63
max_tx_queues : 63
max_mac_addrs : 128
rx_offload_capa : 0xd265f
tx_offload_capa : 0x19faf
rss reta_size : 128
flow_type_rss : 0xc000000000d34
tx_desc_max : 4096
tx_desc_min : 16
rx_desc_max : 8192
rx_desc_min : 16
zmq publisher at: tcp://*:4500
rx_data_q_num : 1
rx_drop_q_num : 0
rx_dp_q_num : 0
rx_que_total : 1
--
rx_desc_num_data_q : 4096
rx_desc_num_drop_q : 4096
rx_desc_num_dp_q : 0
total_desc : 4096
--
tx_desc_num : 1024
port 0 desc: NetXtreme-E Ethernet Virtual Function
rx_qid: 0 (4096)
bnxt_stats_reset_op(): Device Initialization not complete!
bnxt_mac_addr_add_op(): Cannot add MAC address to a VF interface
Failed setting MAC for port 0
Killing Scapy server... Scapy server is killed
Below some other infos:
Open Source Components / Libraries
DPDK (BSD)
YAML-CPP (BSD)
JSONCPP (MIT)
BPF (BSD)
HDR-HISTOGRAM-C (CC0)
Open Source Binaries
ZMQ (LGPL v3plus)
Version : v3.04
DPDK version : DPDK 23.03.0
User : root
Date : Apr 24 2024 , 20:39:54
Uuid : 393570d6-027a-11ef-900d-73f27c253397
Git SHA : 64cd404ed8614ad9d5e5a86f0334100e134647d8
Compiled with GCC : 9.4.0
Compiled with glibc : 2.31 (host: 2.31)
Sanitized image : no
$ sudo ./dpdk_setup_ports.py -t
+----+------+---------+-------------------+--------------------------------------------------------------+----------+-----------+----------+
| ID | NUMA | PCI | MAC | Name | Driver | Linux IF | Active |
+====+======+=========+===================+==============================================================+==========+===========+==========+
| 0 | 0 | 04:00.0 | 2c:ea:7f:b6:8a:1e | NetXtreme BCM5720 2-port Gigabit Ethernet PCIe | tg3 | eno1 | *Active* |
+----+------+---------+-------------------+--------------------------------------------------------------+----------+-----------+----------+
| 1 | 0 | 04:00.1 | 2c:ea:7f:b6:8a:1f | NetXtreme BCM5720 2-port Gigabit Ethernet PCIe | tg3 | eno2 | |
+----+------+---------+-------------------+--------------------------------------------------------------+----------+-----------+----------+
| 2 | 0 | 65:00.0 | b0:26:28:79:e4:20 | BCM57416 NetXtreme-E Dual-Media 10G RDMA Ethernet Controller | bnxt_en | ens2f0np0 | |
+----+------+---------+-------------------+--------------------------------------------------------------+----------+-----------+----------+
| 3 | 0 | 65:00.1 | b0:26:28:79:e4:21 | BCM57416 NetXtreme-E Dual-Media 10G RDMA Ethernet Controller | bnxt_en | ens2f1np1 | |
+----+------+---------+-------------------+--------------------------------------------------------------+----------+-----------+----------+
| 4 | 0 | 65:02.0 | 62:52:b3:8c:eb:0f | NetXtreme-E Ethernet Virtual Function | vfio-pci | | |
+----+------+---------+-------------------+--------------------------------------------------------------+----------+-----------+----------+
| 5 | 0 | 65:03.0 | 8a:82:0e:c2:bf:9e | NetXtreme-E Ethernet Virtual Function | vfio-pci | | |
+----+------+---------+-------------------+--------------------------------------------------------------+----------+-----------+----------+
$ sudo ./dpdk_setup_ports.py -s
Network devices using DPDK-compatible driver
============================================
0000:65:02.0 'NetXtreme-E Ethernet Virtual Function' drv=vfio-pci unused=bnxt_en,igb_uio,uio_pci_generic
0000:65:03.0 'NetXtreme-E Ethernet Virtual Function' drv=vfio-pci unused=bnxt_en,igb_uio,uio_pci_generic
Network devices using kernel driver
===================================
0000:04:00.0 'NetXtreme BCM5720 2-port Gigabit Ethernet PCIe' if=eno1 drv=tg3 unused=igb_uio,vfio-pci,uio_pci_generic *Active*
0000:04:00.1 'NetXtreme BCM5720 2-port Gigabit Ethernet PCIe' if=eno2 drv=tg3 unused=igb_uio,vfio-pci,uio_pci_generic
0000:65:00.0 'BCM57416 NetXtreme-E Dual-Media 10G RDMA Ethernet Controller' if=ens2f0np0 drv=bnxt_en unused=igb_uio,vfio-pci,uio_pci_generic
0000:65:00.1 'BCM57416 NetXtreme-E Dual-Media 10G RDMA Ethernet Controller' if=ens2f1np1 drv=bnxt_en unused=igb_uio,vfio-pci,uio_pci_generic
Other network devices
=====================
<none>
/etc/trex_cfg.yaml
### Config file generated by dpdk_setup_ports.py ###
- version: 2
interfaces: ['65:02.0', '65:03.0']
port_info:
- ip: 10.60.1.45
default_gw: 10.60.1.1
- ip: 10.30.1.45
default_gw: 10.30.1.1
platform:
master_thread_id: 0
latency_thread_id: 15
dual_if:
- socket: 0
threads: [1,2,3,4,5,6,7,8,9,10,11,12,13,14]
lucascgo commented
I tried other versions of TRex and the last version without error in the bxnt driver with the VFs is v2.87.