Cannot Rx after DPDK 22.03 version with iavf(XL710 VF)

Question

Cannot Rx after DPDK 22.03 version with iavf(XL710 VF)

egwakim opened this issue 2 years ago · 11 comments

Hi,
With the latest TRex version.
Sometimes XL710 VF cannot Rx packets when startup.
It happens randomly.
No issues found for several years before applying the latest TRex version.

Used TRex version : The latest git snapshot version of TRex ( master of 22.05.12 version )
Test environment

Used NIC : XL710 ( 40G NIC ) SRIOV-VF NIC
Used driver and F/W version
root:~# ethtool -i eth0
driver: i40e
version: 2.1.14-k
firmware-version: 7.10 0x800075e6 19.5.12
expansion-rom-version:
bus-info: 0000:3b:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes
Reproduction

Start TRex : Sometimes TRex start fails because TRex cannot get ARP from peer because Rx stopped
Broadcasted packets are exist in same subnet
It happens randomly : occurs with 10% ~ 80% probability

Logs

Kernel syslog : Fault found in IOMMU operation with iavf driver, 3b:02.4 stopped working(Could not get ARP)
May 23 03:55:32 xx[31168]: EAL: VFIO support initialized
May 23 03:55:32 xx[31168]: EAL: Using IOMMU type 1 (Type 1)
May 23 03:55:32 kernel: vfio-pci 0000:3b:02.3: enabling device (0000 -> 0002)
May 23 03:55:33 xx[31168]: EAL: Probe PCI driver: net_iavf (8086:154c) device: 0000:3b:02.3 (socket 0)
May 23 03:55:33 kernel: vfio-pci 0000:3b:02.4: enabling device (0000 -> 0002)
May 23 03:55:33 xx[31168]: EAL: Probe PCI driver: net_iavf (8086:154c) device: 0000:3b:02.4 (socket 0)
May 23 03:55:33 xx[31168]: TELEMETRY: No legacy callbacks, legacy socket not created
May 23 03:55:33 xx[31168]: iavf_configure_queues(): RXDID[22] is not supported, request default RXDID[1] in Queue[0]
May 23 03:55:34 kernel: i40e 0000:3b:00.0: Unprivileged VF 3 is attempting to configure promiscuous mode
May 23 03:55:34 kernel: i40e 0000:3b:00.0: Unprivileged VF 3 is attempting to configure promiscuous mode
May 23 03:55:34 xx[31168]: iavf_configure_queues(): RXDID[22] is not supported, request default RXDID[1] in Queue[0]
May 23 03:55:34 kernel: DMAR: DRHD: handling fault status reg 302
May 23 03:55:34 kernel: DMAR: [DMA Write] Request device [3b:02.4] fault addr 0 [fault reason 05] PTE Write access is not set
May 23 03:55:34 kernel: i40e 0000:3b:00.0: Unprivileged VF 4 is attempting to configure promiscuous mode
May 23 03:55:34 kernel: DMAR: [DMA Write] Request device [3b:02.4] fault addr 0 [fault reason 05] PTE Write access is not set
May 23 03:55:34 kernel: DMAR: DRHD: handling fault status reg 500
TRex server log
2022-05-23T03:55:30Z The ports are bound/configured.
2022-05-23T03:55:32Z Starting TRex v2.97 please wait ...
2022-05-23T03:55:32Z Using configuration file /opt/trex_cfg.yaml
2022-05-23T03:55:32Z port limit : 2
2022-05-23T03:55:32Z port_bandwidth_gb : 10
2022-05-23T03:55:32Z port_speed : 0
2022-05-23T03:55:32Z port_mtu : 0
2022-05-23T03:55:32Z if_mask : None
2022-05-23T03:55:32Z prefix : trex1
2022-05-23T03:55:32Z is low-end : 0
2022-05-23T03:55:32Z stack type :
2022-05-23T03:55:32Z limit_memory : 2048
2022-05-23T03:55:32Z thread_per_dual_if : 1
2022-05-23T03:55:32Z if : 0000:3b:02.3, 0000:3b:02.4,
2022-05-23T03:55:32Z enable_zmq_pub : 1
2022-05-23T03:55:32Z zmq_pub_port : 4500
2022-05-23T03:55:32Z m_zmq_rpc_port : 4501
2022-05-23T03:55:32Z m_rx_desc : 4096
2022-05-23T03:55:32Z m_tx_desc : 4096
2022-05-23T03:55:32Z src : 00:00:00:00:00:00
2022-05-23T03:55:32Z dest : 00:00:00:00:00:00
2022-05-23T03:55:32Z src : 00:00:00:00:00:00
2022-05-23T03:55:32Z dest : 00:00:00:00:00:00
2022-05-23T03:55:32Z memory per 2x10G ports
2022-05-23T03:55:32Z MBUF_64 : 16380
2022-05-23T03:55:32Z MBUF_128 : 8190
2022-05-23T03:55:32Z MBUF_256 : 8190
2022-05-23T03:55:32Z MBUF_512 : 8190
2022-05-23T03:55:32Z MBUF_1024 : 8190
2022-05-23T03:55:32Z MBUF_2048 : 4095
2022-05-23T03:55:32Z MBUF_4096 : 128
2022-05-23T03:55:32Z MBUF_9K : 512
2022-05-23T03:55:32Z TRAFFIC_MBUF_64 : 65520
2022-05-23T03:55:32Z TRAFFIC_MBUF_128 : 32760
2022-05-23T03:55:32Z TRAFFIC_MBUF_256 : 8190
2022-05-23T03:55:32Z TRAFFIC_MBUF_512 : 8190
2022-05-23T03:55:32Z TRAFFIC_MBUF_1024 : 8190
2022-05-23T03:55:32Z TRAFFIC_MBUF_2048 : 32760
2022-05-23T03:55:32Z TRAFFIC_MBUF_4096 : 128
2022-05-23T03:55:32Z TRAFFIC_MBUF_9K : 512
2022-05-23T03:55:32Z MBUF_DP_FLOWS : 524288
2022-05-23T03:55:32Z MBUF_GLOBAL_FLOWS : 5120
2022-05-23T03:55:32Z master thread : 0
2022-05-23T03:55:32Z rx thread : 2
2022-05-23T03:55:32Z dual_if : 0
2022-05-23T03:55:32Z socket : 0
2022-05-23T03:55:32Z [ 4 ]
2022-05-23T03:55:32Z CTimerWheelYamlInfo does not exist
2022-05-23T03:55:32Z flags : 8010f00
2022-05-23T03:55:32Z write_file : 0
2022-05-23T03:55:32Z verbose : 7
2022-05-23T03:55:32Z realtime : 1
2022-05-23T03:55:32Z flip : 0
2022-05-23T03:55:32Z cores : 1
2022-05-23T03:55:32Z single core : 0
2022-05-23T03:55:32Z flow-flip : 0
2022-05-23T03:55:32Z no clean close : 0
2022-05-23T03:55:32Z zmq_publish : 1
2022-05-23T03:55:32Z vlan mode : 0
2022-05-23T03:55:32Z client_cfg : 0
2022-05-23T03:55:32Z mbuf_cache_disable : 0
2022-05-23T03:55:32Z cfg file :
2022-05-23T03:55:32Z mac file :
2022-05-23T03:55:32Z out file :
2022-05-23T03:55:32Z client cfg file :
2022-05-23T03:55:32Z duration : 0
2022-05-23T03:55:32Z factor : 1
2022-05-23T03:55:32Z mbuf_factor : 1
2022-05-23T03:55:32Z latency : 0 pkt/sec
2022-05-23T03:55:32Z zmq_port : 4500
2022-05-23T03:55:32Z telnet_port : 4501
2022-05-23T03:55:32Z expected_ports : 2
2022-05-23T03:55:32Z tw_bucket_usec : 20.000000 usec
2022-05-23T03:55:32Z tw_buckets : 1024 usec
2022-05-23T03:55:32Z tw_levels : 3 usec
2022-05-23T03:55:32Z port : 0 dst:00:00:00:00:00:00 src:00:00:00:00:00:00
2022-05-23T03:55:32Z port : 1 dst:00:00:00:00:00:00 src:00:00:00:00:00:00
2022-05-23T03:55:32Z port : 2 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
2022-05-23T03:55:32Z port : 3 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
2022-05-23T03:55:32Z port : 4 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
2022-05-23T03:55:32Z port : 5 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
2022-05-23T03:55:32Z port : 6 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
2022-05-23T03:55:32Z port : 7 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
2022-05-23T03:55:32Z port : 8 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
2022-05-23T03:55:32Z port : 9 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
2022-05-23T03:55:32Z port : 10 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
2022-05-23T03:55:32Z port : 11 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
2022-05-23T03:55:32Z port : 12 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
2022-05-23T03:55:32Z port : 13 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
2022-05-23T03:55:32Z port : 14 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
2022-05-23T03:55:32Z port : 15 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
2022-05-23T03:55:32Z port : 16 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
2022-05-23T03:55:32Z port : 17 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
2022-05-23T03:55:32Z port : 18 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
2022-05-23T03:55:32Z port : 19 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
2022-05-23T03:55:32Z port : 20 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
2022-05-23T03:55:32Z port : 21 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
2022-05-23T03:55:32Z port : 22 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
2022-05-23T03:55:32Z port : 23 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
2022-05-23T03:55:32Z port : 24 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
2022-05-23T03:55:32Z port : 25 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
2022-05-23T03:55:32Z port : 26 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
2022-05-23T03:55:32Z port : 27 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
2022-05-23T03:55:32Z port : 28 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
2022-05-23T03:55:32Z port : 29 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
2022-05-23T03:55:32Z port : 30 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
2022-05-23T03:55:32Z port : 31 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
2022-05-23T03:55:32Z Total Memory :
2022-05-23T03:55:32Z MBUF_64 : 81900
2022-05-23T03:55:32Z MBUF_128 : 40950
2022-05-23T03:55:32Z MBUF_256 : 16380
2022-05-23T03:55:32Z MBUF_512 : 16380
2022-05-23T03:55:32Z MBUF_1024 : 16380
2022-05-23T03:55:32Z MBUF_2048 : 36855
2022-05-23T03:55:32Z MBUF_4096 : 1024
2022-05-23T03:55:32Z MBUF_DP_FLOWS : 524288
2022-05-23T03:55:32Z MBUF_GLOBAL_FLOWS : 5120
2022-05-23T03:55:32Z get_each_core_dp_flows : 524288
2022-05-23T03:55:32Z Total memory : 248.40 Mbytes
2022-05-23T03:55:32Z core_list : 0,2,4
2022-05-23T03:55:32Z sockets : 0
2022-05-23T03:55:32Z active sockets : 1
2022-05-23T03:55:32Z ports_sockets : 1
2022-05-23T03:55:32Z 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
2022-05-23T03:55:32Z phy | virt
2022-05-23T03:55:32Z 4 1
2022-05-23T03:55:32Z DPDK args
2022-05-23T03:55:32Z xx -l 0,2,4 -n 4 --log-level 8 --main-lcore 0 -a 0000:3b:02.3 -a 0000:3b:02.4 --file-prefix trex1 --socket-mem 2048,0
2022-05-23T03:55:32Z EAL: Detected CPU lcores: 80
2022-05-23T03:55:32Z EAL: Detected NUMA nodes: 2
2022-05-23T03:55:32Z EAL: Detected static linkage of DPDK
2022-05-23T03:55:32Z
2022-05-23T03:55:32Z EAL: Multi-process socket /var/run/dpdk/trex1/mp_socket
2022-05-23T03:55:32Z EAL: Selected IOVA mode 'VA'
2022-05-23T03:55:32Z EAL: VFIO support initialized
2022-05-23T03:55:33Z EAL: Using IOMMU type 1 (Type 1)
2022-05-23T03:55:33Z EAL: Probe PCI driver: net_iavf (8086:154c) device: 0000:3b:02.3 (socket 0)
2022-05-23T03:55:33Z EAL: Probe PCI driver: net_iavf (8086:154c) device: 0000:3b:02.4 (socket 0)
2022-05-23T03:55:33Z TELEMETRY: No legacy callbacks, legacy socket not created
2022-05-23T03:55:33Z input : [0000:3b:02.3, 0000:3b:02.4]
2022-05-23T03:55:33Z dpdk : [0000:3b:02.3, 0000:3b:02.4]
2022-05-23T03:55:33Z pci_scan : [0000:3b:02.3, 0000:3b:02.4]
2022-05-23T03:55:33Z map : [ 0, 1]
2022-05-23T03:55:33Z TRex port mapping
2022-05-23T03:55:33Z -----------------
2022-05-23T03:55:33Z TRex vport: 0 dpdk_rte_eth: 0
2022-05-23T03:55:33Z TRex vport: 1 dpdk_rte_eth: 1
2022-05-23T03:55:33Z set driver name net_iavf
2022-05-23T03:55:33Z driver capability : TCP_UDP_OFFLOAD TSO SLRO
2022-05-23T03:55:33Z Warning TSO is supported and asked to be disabled by user
2022-05-23T03:55:33Z Warning SLRO is supported and asked to be disabled by user
2022-05-23T03:55:33Z set dpdk queues mode to ONE_QUE
2022-05-23T03:55:33Z DPDK devices 2 : 2
2022-05-23T03:55:33Z -----
2022-05-23T03:55:33Z 0 : vdev 0000:3b:02.3
2022-05-23T03:55:33Z 1 : vdev 0000:3b:02.4
2022-05-23T03:55:33Z -----
2022-05-23T03:55:33Z Number of ports found: 2
2022-05-23T03:55:33Z
2022-05-23T03:55:33Z
2022-05-23T03:55:33Z if_index : 0
2022-05-23T03:55:33Z driver name : net_iavf
2022-05-23T03:55:33Z min_rx_bufsize : 1024
2022-05-23T03:55:33Z max_rx_pktlen : 9728
2022-05-23T03:55:33Z max_rx_queues : 256
2022-05-23T03:55:33Z max_tx_queues : 256
2022-05-23T03:55:33Z max_mac_addrs : 64
2022-05-23T03:55:33Z rx_offload_capa : 0x8226f
2022-05-23T03:55:33Z tx_offload_capa : 0x19fbf
2022-05-23T03:55:33Z rss reta_size : 64
2022-05-23T03:55:33Z flow_type_rss : 0x3ffc
2022-05-23T03:55:33Z tx_desc_max : 4096
2022-05-23T03:55:33Z tx_desc_min : 64
2022-05-23T03:55:33Z rx_desc_max : 4096
2022-05-23T03:55:33Z rx_desc_min : 64
2022-05-23T03:55:33Z zmq publisher at: tcp://*:4500
2022-05-23T03:55:33Z rx_data_q_num : 1
2022-05-23T03:55:33Z rx_drop_q_num : 0
2022-05-23T03:55:33Z rx_dp_q_num : 0
2022-05-23T03:55:33Z rx_que_total : 1
2022-05-23T03:55:33Z --
2022-05-23T03:55:33Z rx_desc_num_data_q : 4096
2022-05-23T03:55:33Z rx_desc_num_drop_q : 4096
2022-05-23T03:55:33Z rx_desc_num_dp_q : 4096
2022-05-23T03:55:33Z total_desc : 4096
2022-05-23T03:55:33Z --
2022-05-23T03:55:33Z tx_desc_num : 4096
2022-05-23T03:55:33Z port 0 desc: Ethernet Virtual Function 700 Series
2022-05-23T03:55:33Z rx_qid: 0 (4096)
2022-05-23T03:55:33Z iavf_configure_queues(): RXDID[22] is not supported, request default RXDID[1] in Queue[0]
2022-05-23T03:55:34Z port 1 desc: Ethernet Virtual Function 700 Series
2022-05-23T03:55:34Z rx_qid: 0 (4096)
2022-05-23T03:55:34Z iavf_configure_queues(): RXDID[22] is not supported, request default RXDID[1] in Queue[0]
2022-05-23T03:55:35Z wait 1 sec
2022-05-23T03:55:36Z .
2022-05-23T03:55:36Z port : 0
2022-05-23T03:55:36Z ------------
2022-05-23T03:55:36Z link : link : Link Up - speed 25000 Mbps - full-duplex
2022-05-23T03:55:36Z promiscuous : 0
2022-05-23T03:55:36Z port : 1
2022-05-23T03:55:36Z ------------
2022-05-23T03:55:36Z link : link : Link Up - speed 25000 Mbps - full-duplex
2022-05-23T03:55:36Z promiscuous : 0
2022-05-23T03:55:36Z number of ports : 2
2022-05-23T03:55:36Z max cores for 2 ports : 1
2022-05-23T03:55:36Z tx queues per port : 3
2022-05-23T03:55:36Z -------------------------------
2022-05-23T03:55:36Z RX core uses TX queue number 65535 on all ports
2022-05-23T03:55:36Z core, c-port, c-queue, s-port, s-queue, lat-queue
2022-05-23T03:55:36Z ------------------------------------------
2022-05-23T03:55:36Z 1 0 0 1 0 0
2022-05-23T03:55:36Z -------------------------------
2022-05-23T03:55:36Z base stack ctor
2022-05-23T03:55:36Z Legacy stack ctor
2022-05-23T03:55:36Z add port node
2022-05-23T03:55:36Z add node
2022-05-23T03:55:36Z Run pending tasks for ticket 0
2022-05-23T03:55:36Z Running in FG
2022-05-23T03:55:36Z Legacy node ctor
2022-05-23T03:55:36Z base stack ctor
2022-05-23T03:55:36Z Legacy stack ctor
2022-05-23T03:55:36Z add port node
2022-05-23T03:55:36Z add node
2022-05-23T03:55:36Z Run pending tasks for ticket 0
2022-05-23T03:55:36Z Running in FG
2022-05-23T03:55:36Z Legacy node ctor
2022-05-23T03:55:36Z TX grat ARP on port 0 - ip: 10.255.6.131 mac: fe:ee:d6:2a:c8:15
2022-05-23T03:55:36Z TX grat ARP on port 1 - ip: 10.255.6.132 mac: 2e:8e:09:fe:1f:0b
2022-05-23T03:55:36Z TX ARP request on port 0 - ip: 10.255.6.133 mac: Unknown
2022-05-23T03:55:36Z
2022-05-23T03:55:36Z +---------+---------------+----------+
2022-05-23T03:55:36Z 00:00:00,000,000 ETHER
2022-05-23T03:55:36Z |0 |ff|ff|ff|ff|ff|ff|fe|ee|d6|2a|c8|15|08|06|00|01|08|00|06|04|00|01|fe|ee|d6|2a|c8|15|0a|ff|06|83|01|03|05|07|09|00|0a|ff|06|85|00|00|00|00|00|00|00|00|00|00|00|00|00|00|00|00|00|00|
2022-05-23T03:55:36Z TX ARP request on port 1 - ip: 10.255.6.133 mac: Unknown
2022-05-23T03:55:36Z
2022-05-23T03:55:36Z +---------+---------------+----------+
2022-05-23T03:55:36Z 00:00:00,000,000 ETHER
2022-05-23T03:55:36Z |0 |ff|ff|ff|ff|ff|ff|2e|8e|09|fe|1f|0b|08|06|00|01|08|00|06|04|00|01|2e|8e|09|fe|1f|0b|0a|ff|06|84|01|03|05|07|09|01|0a|ff|06|85|00|00|00|00|00|00|00|00|00|00|00|00|00|00|00|00|00|00|
2022-05-23T03:55:36Z RX-gen on port 0 queue 0
2022-05-23T03:55:36Z
2022-05-23T03:55:36Z +---------+---------------+----------+
2022-05-23T03:55:36Z 00:00:00,000,000 ETHER
2022-05-23T03:55:36Z |0 |ff|ff|ff|ff|ff|ff|e4|5d|37|09|bf|99|08|06|00|01|08|00|06|04|00|01|e4|5d|37|09|bf|99|0a|ff|00|02|00|00|00|00|00|00|0a|ff|06|a8|cb|b9|6b|28|00|00|00|00|00|00|00|00|00|00|00|00|00|00|
2022-05-23T03:55:36Z RX ARP request on port 0 queue 0 sip:10.255.0.2 tip:10.255.6.168 vlan:0
2022-05-23T03:55:36Z
2022-05-23T03:55:36Z +---------+---------------+----------+
2022-05-23T03:55:36Z 00:00:00,000,000 ETHER
2022-05-23T03:55:36Z |0 |ff|ff|ff|ff|ff|ff|e4|5d|37|09|bf|99|08|06|00|01|08|00|06|04|00|01|e4|5d|37|09|bf|99|0a|ff|00|02|00|00|00|00|00|00|0a|ff|06|a8|cb|b9|6b|28|00|00|00|00|00|00|00|00|00|00|00|00|00|00|
2022-05-23T03:55:36Z RX-gen on port 0 queue 0
2022-05-23T03:55:36Z
2022-05-23T03:55:36Z +---------+---------------+----------+
2022-05-23T03:55:36Z 00:00:00,000,000 ETHER
2022-05-23T03:55:36Z |0 |ff|ff|ff|ff|ff|ff|e4|5d|37|09|bf|99|08|06|00|01|08|00|06|04|00|01|e4|5d|37|09|bf|99|0a|ff|00|02|00|00|00|00|00|00|0a|ff|06|40|74|8f|f6|4b|00|00|00|00|00|00|00|00|00|00|00|00|00|00|
2022-05-23T03:55:36Z RX ARP request on port 0 queue 0 sip:10.255.0.2 tip:10.255.6.64 vlan:0
2022-05-23T03:55:36Z
2022-05-23T03:55:36Z +---------+---------------+----------+
2022-05-23T03:55:36Z 00:00:00,000,000 ETHER
2022-05-23T03:55:36Z |0 |ff|ff|ff|ff|ff|ff|e4|5d|37|09|bf|99|08|06|00|01|08|00|06|04|00|01|e4|5d|37|09|bf|99|0a|ff|00|02|00|00|00|00|00|00|0a|ff|06|40|74|8f|f6|4b|00|00|00|00|00|00|00|00|00|00|00|00|00|00|
2022-05-23T03:55:36Z RX-gen on port 0 queue 0
2022-05-23T03:55:36Z
2022-05-23T03:55:36Z +---------+---------------+----------+
2022-05-23T03:55:36Z 00:00:00,000,000 ETHER
2022-05-23T03:55:36Z |0 |ff|ff|ff|ff|ff|ff|2e|8e|09|fe|1f|0b|08|06|00|01|08|00|06|04|00|01|2e|8e|09|fe|1f|0b|0a|ff|06|84|01|03|05|07|09|00|0a|ff|06|84|00|00|00|00|00|00|00|00|00|00|00|00|00|00|00|00|00|00|
2022-05-23T03:55:36Z RX grat ARP on port 0 queue 0 sip:10.255.6.132 tip:10.255.6.132 vlan:0
2022-05-23T03:55:36Z
2022-05-23T03:55:36Z +---------+---------------+----------+
2022-05-23T03:55:36Z 00:00:00,000,000 ETHER
2022-05-23T03:55:36Z |0 |ff|ff|ff|ff|ff|ff|2e|8e|09|fe|1f|0b|08|06|00|01|08|00|06|04|00|01|2e|8e|09|fe|1f|0b|0a|ff|06|84|01|03|05|07|09|00|0a|ff|06|84|00|00|00|00|00|00|00|00|00|00|00|00|00|00|00|00|00|00|
2022-05-23T03:55:36Z RX-gen on port 0 queue 0
2022-05-23T03:55:36Z
2022-05-23T03:55:36Z +---------+---------------+----------+
2022-05-23T03:55:36Z 00:00:00,000,000 ETHER
2022-05-23T03:55:36Z |0 |ff|ff|ff|ff|ff|ff|2e|8e|09|fe|1f|0b|08|06|00|01|08|00|06|04|00|01|2e|8e|09|fe|1f|0b|0a|ff|06|84|01|03|05|07|09|01|0a|ff|06|85|00|00|00|00|00|00|00|00|00|00|00|00|00|00|00|00|00|00|
2022-05-23T03:55:36Z RX ARP request on port 0 queue 0 sip:10.255.6.132 tip:10.255.6.133 vlan:0
2022-05-23T03:55:36Z
2022-05-23T03:55:36Z +---------+---------------+----------+
2022-05-23T03:55:36Z 00:00:00,000,000 ETHER
2022-05-23T03:55:36Z |0 |ff|ff|ff|ff|ff|ff|2e|8e|09|fe|1f|0b|08|06|00|01|08|00|06|04|00|01|2e|8e|09|fe|1f|0b|0a|ff|06|84|01|03|05|07|09|01|0a|ff|06|85|00|00|00|00|00|00|00|00|00|00|00|00|00|00|00|00|00|00|
2022-05-23T03:55:36Z RX-gen on port 0 queue 0
2022-05-23T03:55:36Z
2022-05-23T03:55:36Z +---------+---------------+----------+
2022-05-23T03:55:36Z 00:00:00,000,000 ETHER
2022-05-23T03:55:36Z |0 |fe|ee|d6|2a|c8|15|0a|35|72|45|2e|5f|08|06|00|01|08|00|06|04|00|02|0a|35|72|45|2e|5f|0a|ff|06|85|fe|ee|d6|2a|c8|15|0a|ff|06|83|00|00|00|00|00|00|00|00|00|00|00|00|00|00|00|00|00|00|
2022-05-23T03:55:36Z RX ARP reply on port 0 queue 0 sip:10.255.6.133 tip:10.255.6.131 vlan:0
2022-05-23T03:55:36Z RX-gen on port 0 queue 0
2022-05-23T03:55:36Z
2022-05-23T03:55:36Z +---------+---------------+----------+
2022-05-23T03:55:36Z 00:00:00,000,000 ETHER
2022-05-23T03:55:36Z |0 |ff|ff|ff|ff|ff|ff|e4|5d|37|0f|ef|99|08|06|00|01|08|00|06|04|00|01|e4|5d|37|0f|ef|99|0a|ff|00|03|00|00|00|00|00|00|0a|ff|06|0f|50|4c|1f|ea|00|00|00|00|00|00|00|00|00|00|00|00|00|00|
2022-05-23T03:55:36Z RX ARP request on port 0 queue 0 sip:10.255.0.3 tip:10.255.6.15 vlan:0
2022-05-23T03:55:36Z
2022-05-23T03:55:36Z +---------+---------------+----------+
2022-05-23T03:55:36Z 00:00:00,000,000 ETHER
2022-05-23T03:55:36Z |0 |ff|ff|ff|ff|ff|ff|e4|5d|37|0f|ef|99|08|06|00|01|08|00|06|04|00|01|e4|5d|37|0f|ef|99|0a|ff|00|03|00|00|00|00|00|00|0a|ff|06|0f|50|4c|1f|ea|00|00|00|00|00|00|00|00|00|00|00|00|00|00|
2022-05-23T03:55:36Z RX-gen on port 0 queue 0
2022-05-23T03:55:36Z
2022-05-23T03:55:36Z +---------+---------------+----------+
2022-05-23T03:55:36Z 00:00:00,000,000 ETHER
2022-05-23T03:55:36Z |0 |ff|ff|ff|ff|ff|ff|e4|5d|37|0f|ef|99|08|06|00|01|08|00|06|04|00|01|e4|5d|37|0f|ef|99|0a|ff|00|03|00|00|00|00|00|00|0a|ff|06|a6|bd|68|0a|28|00|00|00|00|00|00|00|00|00|00|00|00|00|00|
2022-05-23T03:55:36Z RX ARP request on port 0 queue 0 sip:10.255.0.3 tip:10.255.6.166 vlan:0
2022-05-23T03:55:36Z
2022-05-23T03:55:36Z +---------+---------------+----------+
2022-05-23T03:55:36Z 00:00:00,000,000 ETHER
2022-05-23T03:55:36Z |0 |ff|ff|ff|ff|ff|ff|e4|5d|37|0f|ef|99|08|06|00|01|08|00|06|04|00|01|e4|5d|37|0f|ef|99|0a|ff|00|03|00|00|00|00|00|00|0a|ff|06|a6|bd|68|0a|28|00|00|00|00|00|00|00|00|00|00|00|00|00|00|
2022-05-23T03:55:36Z RX-gen on port 0 queue 0
2022-05-23T03:55:36Z
2022-05-23T03:55:36Z +---------+---------------+----------+
2022-05-23T03:55:36Z 00:00:00,000,000 ETHER
2022-05-23T03:55:36Z |0 |ff|ff|ff|ff|ff|ff|e4|5d|37|0f|ef|99|08|06|00|01|08|00|06|04|00|01|e4|5d|37|0f|ef|99|0a|ff|00|03|00|00|00|00|00|00|0a|ff|07|e6|83|5d|64|a5|00|00|00|00|00|00|00|00|00|00|00|00|00|00|
2022-05-23T03:55:36Z RX ARP request on port 0 queue 0 sip:10.255.0.3 tip:10.255.7.230 vlan:0
2022-05-23T03:55:36Z
2022-05-23T03:55:36Z +---------+---------------+----------+
2022-05-23T03:55:36Z 00:00:00,000,000 ETHER
2022-05-23T03:55:36Z |0 |ff|ff|ff|ff|ff|ff|e4|5d|37|0f|ef|99|08|06|00|01|08|00|06|04|00|01|e4|5d|37|0f|ef|99|0a|ff|00|03|00|00|00|00|00|00|0a|ff|07|e6|83|5d|64|a5|00|00|00|00|00|00|00|00|00|00|00|00|00|00|
2022-05-23T03:55:36Z RX-gen on port 0 queue 0
2022-05-23T03:55:36Z
2022-05-23T03:55:36Z +---------+---------------+----------+
2022-05-23T03:55:36Z 00:00:00,000,000 ETHER
2022-05-23T03:55:36Z |0 |ff|ff|ff|ff|ff|ff|e4|5d|37|09|bf|99|08|06|00|01|08|00|06|04|00|01|e4|5d|37|09|bf|99|0a|ff|00|02|00|00|00|00|00|00|0a|ff|06|8c|a7|92|de|f9|00|00|00|00|00|00|00|00|00|00|00|00|00|00|
2022-05-23T03:55:36Z RX ARP request on port 0 queue 0 sip:10.255.0.2 tip:10.255.6.140 vlan:0
2022-05-23T03:55:36Z
2022-05-23T03:55:36Z +---------+---------------+----------+
2022-05-23T03:55:36Z 00:00:00,000,000 ETHER
2022-05-23T03:55:36Z |0 |ff|ff|ff|ff|ff|ff|e4|5d|37|09|bf|99|08|06|00|01|08|00|06|04|00|01|e4|5d|37|09|bf|99|0a|ff|00|02|00|00|00|00|00|00|0a|ff|06|8c|a7|92|de|f9|00|00|00|00|00|00|00|00|00|00|00|00|00|00|
2022-05-23T03:55:36Z RX-gen on port 0 queue 0
2022-05-23T03:55:36Z
2022-05-23T03:55:36Z +---------+---------------+----------+
2022-05-23T03:55:36Z 00:00:00,000,000 ETHER
2022-05-23T03:55:36Z |0 |ff|ff|ff|ff|ff|ff|e4|5d|37|09|bf|99|08|06|00|01|08|00|06|04|00|01|e4|5d|37|09|bf|99|0a|ff|00|02|00|00|00|00|00|00|0a|ff|06|0d|e0|39|f6|9e|00|00|00|00|00|00|00|00|00|00|00|00|00|00|
2022-05-23T03:55:36Z RX ARP request on port 0 queue 0 sip:10.255.0.2 tip:10.255.6.13 vlan:0
2022-05-23T03:55:36Z
2022-05-23T03:55:36Z +---------+---------------+----------+
2022-05-23T03:55:36Z 00:00:00,000,000 ETHER
2022-05-23T03:55:36Z |0 |ff|ff|ff|ff|ff|ff|e4|5d|37|09|bf|99|08|06|00|01|08|00|06|04|00|01|e4|5d|37|09|bf|99|0a|ff|00|02|00|00|00|00|00|00|0a|ff|06|0d|e0|39|f6|9e|00|00|00|00|00|00|00|00|00|00|00|00|00|00|
2022-05-23T03:55:36Z RX-gen on port 0 queue 0
2022-05-23T03:55:36Z
2022-05-23T03:55:36Z +---------+---------------+----------+
2022-05-23T03:55:36Z 00:00:00,000,000 ETHER
2022-05-23T03:55:36Z |0 |ff|ff|ff|ff|ff|ff|e4|5d|37|09|bf|99|08|06|00|01|08|00|06|04|00|01|e4|5d|37|09|bf|99|0a|ff|00|02|00|00|00|00|00|00|0a|ff|06|0e|ab|53|65|bf|00|00|00|00|00|00|00|00|00|00|00|00|00|00|
2022-05-23T03:55:36Z RX ARP request on port 0 queue 0 sip:10.255.0.2 tip:10.255.6.14 vlan:0
2022-05-23T03:55:36Z
2022-05-23T03:55:36Z +---------+---------------+----------+
2022-05-23T03:55:36Z 00:00:00,000,000 ETHER
2022-05-23T03:55:36Z |0 |ff|ff|ff|ff|ff|ff|e4|5d|37|09|bf|99|08|06|00|01|08|00|06|04|00|01|e4|5d|37|09|bf|99|0a|ff|00|02|00|00|00|00|00|00|0a|ff|06|0e|ab|53|65|bf|00|00|00|00|00|00|00|00|00|00|00|00|00|00|
2022-05-23T03:55:36Z RX-gen on port 0 queue 0
2022-05-23T03:55:36Z
2022-05-23T03:55:36Z +---------+---------------+----------+
2022-05-23T03:55:36Z 00:00:00,000,000 ETHER
2022-05-23T03:55:36Z |0 |ff|ff|ff|ff|ff|ff|e4|5d|37|0f|ef|99|08|06|00|01|08|00|06|04|00|01|e4|5d|37|0f|ef|99|0a|ff|00|03|00|00|00|00|00|00|0a|ff|06|41|73|56|95|ec|00|00|00|00|00|00|00|00|00|00|00|00|00|00|
2022-05-23T03:55:36Z RX ARP request on port 0 queue 0 sip:10.255.0.3 tip:10.255.6.65 vlan:0
2022-05-23T03:55:36Z
2022-05-23T03:55:36Z +---------+---------------+----------+
2022-05-23T03:55:36Z 00:00:00,000,000 ETHER
2022-05-23T03:55:36Z |0 |ff|ff|ff|ff|ff|ff|e4|5d|37|0f|ef|99|08|06|00|01|08|00|06|04|00|01|e4|5d|37|0f|ef|99|0a|ff|00|03|00|00|00|00|00|00|0a|ff|06|41|73|56|95|ec|00|00|00|00|00|00|00|00|00|00|00|00|00|00|
2022-05-23T03:55:36Z RX-gen on port 0 queue 0
2022-05-23T03:55:36Z
2022-05-23T03:55:36Z +---------+---------------+----------+
2022-05-23T03:55:36Z 00:00:00,000,000 ETHER
2022-05-23T03:55:36Z |0 |ff|ff|ff|ff|ff|ff|e4|5d|37|0f|ef|99|08|06|00|01|08|00|06|04|00|01|e4|5d|37|0f|ef|99|0a|ff|00|03|00|00|00|00|00|00|0a|ff|06|0d|7f|06|c2|47|00|00|00|00|00|00|00|00|00|00|00|00|00|00|
2022-05-23T03:55:36Z RX ARP request on port 0 queue 0 sip:10.255.0.3 tip:10.255.6.13 vlan:0
2022-05-23T03:55:36Z
2022-05-23T03:55:36Z +---------+---------------+----------+
2022-05-23T03:55:36Z 00:00:00,000,000 ETHER
2022-05-23T03:55:36Z |0 |ff|ff|ff|ff|ff|ff|e4|5d|37|0f|ef|99|08|06|00|01|08|00|06|04|00|01|e4|5d|37|0f|ef|99|0a|ff|00|03|00|00|00|00|00|00|0a|ff|06|0d|7f|06|c2|47|00|00|00|00|00|00|00|00|00|00|00|00|00|00|
2022-05-23T03:55:36Z RX-gen on port 0 queue 0
2022-05-23T03:55:36Z
2022-05-23T03:55:36Z +---------+---------------+----------+
2022-05-23T03:55:36Z 00:00:00,000,000 ETHER
2022-05-23T03:55:36Z |0 |ff|ff|ff|ff|ff|ff|e4|5d|37|09|bf|99|08|06|00|01|08|00|06|04|00|01|e4|5d|37|09|bf|99|0a|ff|00|02|00|00|00|00|00|00|0a|ff|06|41|82|65|35|0e|00|00|00|00|00|00|00|00|00|00|00|00|00|00|
2022-05-23T03:55:36Z RX ARP request on port 0 queue 0 sip:10.255.0.2 tip:10.255.6.65 vlan:0
2022-05-23T03:55:36Z
2022-05-23T03:55:36Z +---------+---------------+----------+
2022-05-23T03:55:36Z 00:00:00,000,000 ETHER
2022-05-23T03:55:36Z |0 |ff|ff|ff|ff|ff|ff|e4|5d|37|09|bf|99|08|06|00|01|08|00|06|04|00|01|e4|5d|37|09|bf|99|0a|ff|00|02|00|00|00|00|00|00|0a|ff|06|41|82|65|35|0e|00|00|00|00|00|00|00|00|00|00|00|00|00|00|
2022-05-23T03:55:36Z RX-gen on port 0 queue 0
2022-05-23T03:55:36Z
2022-05-23T03:55:36Z +---------+---------------+----------+
2022-05-23T03:55:36Z 00:00:00,000,000 ETHER
2022-05-23T03:55:36Z |0 |ff|ff|ff|ff|ff|ff|e4|5d|37|09|bf|99|08|06|00|01|08|00|06|04|00|01|e4|5d|37|09|bf|99|0a|ff|00|02|00|00|00|00|00|00|0a|ff|06|a6|09|c1|ac|a8|00|00|00|00|00|00|00|00|00|00|00|00|00|00|
2022-05-23T03:55:36Z RX ARP request on port 0 queue 0 sip:10.255.0.2 tip:10.255.6.166 vlan:0
2022-05-23T03:55:36Z
2022-05-23T03:55:36Z +---------+---------------+----------+
2022-05-23T03:55:36Z 00:00:00,000,000 ETHER
2022-05-23T03:55:36Z |0 |ff|ff|ff|ff|ff|ff|e4|5d|37|09|bf|99|08|06|00|01|08|00|06|04|00|01|e4|5d|37|09|bf|99|0a|ff|00|02|00|00|00|00|00|00|0a|ff|06|a6|09|c1|ac|a8|00|00|00|00|00|00|00|00|00|00|00|00|00|00|
2022-05-23T03:55:36Z RX-gen on port 0 queue 0
2022-05-23T03:55:36Z
2022-05-23T03:55:36Z +---------+---------------+----------+
2022-05-23T03:55:36Z 00:00:00,000,000 ETHER
2022-05-23T03:55:36Z |0 |ff|ff|ff|ff|ff|ff|e4|5d|37|0f|ef|99|08|06|00|01|08|00|06|04|00|01|e4|5d|37|0f|ef|99|0a|ff|00|03|00|00|00|00|00|00|0a|ff|06|ae|8d|1f|9e|f7|00|00|00|00|00|00|00|00|00|00|00|00|00|00|
2022-05-23T03:55:36Z RX ARP request on port 0 queue 0 sip:10.255.0.3 tip:10.255.6.174 vlan:0
2022-05-23T03:55:36Z
2022-05-23T03:55:36Z +---------+---------------+----------+
2022-05-23T03:55:36Z 00:00:00,000,000 ETHER
2022-05-23T03:55:36Z |0 |ff|ff|ff|ff|ff|ff|e4|5d|37|0f|ef|99|08|06|00|01|08|00|06|04|00|01|e4|5d|37|0f|ef|99|0a|ff|00|03|00|00|00|00|00|00|0a|ff|06|ae|8d|1f|9e|f7|00|00|00|00|00|00|00|00|00|00|00|00|00|00|
2022-05-23T03:55:36Z RX-gen on port 0 queue 0
2022-05-23T03:55:36Z
2022-05-23T03:55:36Z +---------+---------------+----------+
2022-05-23T03:55:36Z 00:00:00,000,000 ETHER
2022-05-23T03:55:36Z |0 |ff|ff|ff|ff|ff|ff|e4|5d|37|09|bf|99|08|06|00|01|08|00|06|04|00|01|e4|5d|37|09|bf|99|0a|ff|00|02|00|00|00|00|00|00|0a|ff|06|0f|fa|4c|c0|b5|00|00|00|00|00|00|00|00|00|00|00|00|00|00|
2022-05-23T03:55:36Z RX ARP request on port 0 queue 0 sip:10.255.0.2 tip:10.255.6.15 vlan:0
2022-05-23T03:55:36Z
2022-05-23T03:55:36Z +---------+---------------+----------+
2022-05-23T03:55:36Z 00:00:00,000,000 ETHER
2022-05-23T03:55:36Z |0 |ff|ff|ff|ff|ff|ff|e4|5d|37|09|bf|99|08|06|00|01|08|00|06|04|00|01|e4|5d|37|09|bf|99|0a|ff|00|02|00|00|00|00|00|00|0a|ff|06|0f|fa|4c|c0|b5|00|00|00|00|00|00|00|00|00|00|00|00|00|00|
2022-05-23T03:55:36Z RX-gen on port 0 queue 0
2022-05-23T03:55:36Z
2022-05-23T03:55:36Z +---------+---------------+----------+
2022-05-23T03:55:36Z 00:00:00,000,000 ETHER
2022-05-23T03:55:36Z |0 |ff|ff|ff|ff|ff|ff|e4|5d|37|0f|ef|99|08|06|00|01|08|00|06|04|00|01|e4|5d|37|0f|ef|99|0a|ff|00|03|00|00|00|00|00|00|0a|ff|06|40|27|43|41|98|00|00|00|00|00|00|00|00|00|00|00|00|00|00|
2022-05-23T03:55:36Z RX ARP request on port 0 queue 0 sip:10.255.0.3 tip:10.255.6.64 vlan:0
2022-05-23T03:55:36Z
2022-05-23T03:55:36Z +---------+---------------+----------+
2022-05-23T03:55:36Z 00:00:00,000,000 ETHER
2022-05-23T03:55:36Z |0 |ff|ff|ff|ff|ff|ff|e4|5d|37|0f|ef|99|08|06|00|01|08|00|06|04|00|01|e4|5d|37|0f|ef|99|0a|ff|00|03|00|00|00|00|00|00|0a|ff|06|40|27|43|41|98|00|00|00|00|00|00|00|00|00|00|00|00|00|00|
2022-05-23T03:55:36Z RX-gen on port 0 queue 0
2022-05-23T03:55:36Z
2022-05-23T03:55:36Z +---------+---------------+----------+
2022-05-23T03:55:36Z 00:00:00,000,000 ETHER
2022-05-23T03:55:36Z |0 |ff|ff|ff|ff|ff|ff|e4|5d|37|09|bf|99|08|06|00|01|08|00|06|04|00|01|e4|5d|37|09|bf|99|0a|ff|00|02|00|00|00|00|00|00|0a|ff|06|ae|c2|63|db|f5|00|00|00|00|00|00|00|00|00|00|00|00|00|00|
2022-05-23T03:55:36Z RX ARP request on port 0 queue 0 sip:10.255.0.2 tip:10.255.6.174 vlan:0
2022-05-23T03:55:36Z
2022-05-23T03:55:36Z +---------+---------------+----------+
2022-05-23T03:55:36Z 00:00:00,000,000 ETHER
2022-05-23T03:55:36Z |0 |ff|ff|ff|ff|ff|ff|e4|5d|37|09|bf|99|08|06|00|01|08|00|06|04|00|01|e4|5d|37|09|bf|99|0a|ff|00|02|00|00|00|00|00|00|0a|ff|06|ae|c2|63|db|f5|00|00|00|00|00|00|00|00|00|00|00|00|00|00|
2022-05-23T03:55:36Z RX-gen on port 0 queue 0
2022-05-23T03:55:36Z
2022-05-23T03:55:36Z +---------+---------------+----------+
2022-05-23T03:55:36Z 00:00:00,000,000 ETHER
2022-05-23T03:55:36Z |0 |ff|ff|ff|ff|ff|ff|e4|5d|37|09|bf|99|08|06|00|01|08|00|06|04|00|01|e4|5d|37|09|bf|99|0a|ff|00|02|00|00|00|00|00|00|0a|ff|06|40|3d|ce|a8|60|00|00|00|00|00|00|00|00|00|00|00|00|00|00|
2022-05-23T03:55:36Z RX ARP request on port 0 queue 0 sip:10.255.0.2 tip:10.255.6.64 vlan:0
2022-05-23T03:55:36Z
2022-05-23T03:55:36Z +---------+---------------+----------+
2022-05-23T03:55:36Z 00:00:00,000,000 ETHER
2022-05-23T03:55:36Z |0 |ff|ff|ff|ff|ff|ff|e4|5d|37|09|bf|99|08|06|00|01|08|00|06|04|00|01|e4|5d|37|09|bf|99|0a|ff|00|02|00|00|00|00|00|00|0a|ff|06|40|3d|ce|a8|60|00|00|00|00|00|00|00|00|00|00|00|00|00|00|
2022-05-23T03:55:36Z RX-gen on port 0 queue 0
2022-05-23T03:55:36Z
2022-05-23T03:55:36Z +---------+---------------+----------+
2022-05-23T03:55:36Z 00:00:00,000,000 ETHER
2022-05-23T03:55:36Z |0 |ff|ff|ff|ff|ff|ff|e4|5d|37|0f|ef|99|08|06|00|01|08|00|06|04|00|01|e4|5d|37|0f|ef|99|0a|ff|00|03|00|00|00|00|00|00|0a|ff|06|8c|4f|78|80|3c|00|00|00|00|00|00|00|00|00|00|00|00|00|00|
2022-05-23T03:55:36Z RX ARP request on port 0 queue 0 sip:10.255.0.3 tip:10.255.6.140 vlan:0
2022-05-23T03:55:36Z
2022-05-23T03:55:36Z +---------+---------------+----------+
2022-05-23T03:55:36Z 00:00:00,000,000 ETHER
2022-05-23T03:55:36Z |0 |ff|ff|ff|ff|ff|ff|e4|5d|37|0f|ef|99|08|06|00|01|08|00|06|04|00|01|e4|5d|37|0f|ef|99|0a|ff|00|03|00|00|00|00|00|00|0a|ff|06|8c|4f|78|80|3c|00|00|00|00|00|00|00|00|00|00|00|00|00|00|
2022-05-23T03:55:36Z RX-gen on port 0 queue 0
2022-05-23T03:55:36Z
2022-05-23T03:55:36Z +---------+---------------+----------+
2022-05-23T03:55:36Z 00:00:00,000,000 ETHER
2022-05-23T03:55:36Z |0 |ff|ff|ff|ff|ff|ff|e4|5d|37|0f|ef|99|08|06|00|01|08|00|06|04|00|01|e4|5d|37|0f|ef|99|0a|ff|00|03|00|00|00|00|00|00|0a|ff|06|a8|f7|51|fc|c6|00|00|00|00|00|00|00|00|00|00|00|00|00|00|
2022-05-23T03:55:36Z RX ARP request on port 0 queue 0 sip:10.255.0.3 tip:10.255.6.168 vlan:0
2022-05-23T03:55:36Z
2022-05-23T03:55:36Z +---------+---------------+----------+
2022-05-23T03:55:36Z 00:00:00,000,000 ETHER
2022-05-23T03:55:36Z |0 |ff|ff|ff|ff|ff|ff|e4|5d|37|0f|ef|99|08|06|00|01|08|00|06|04|00|01|e4|5d|37|0f|ef|99|0a|ff|00|03|00|00|00|00|00|00|0a|ff|06|a8|f7|51|fc|c6|00|00|00|00|00|00|00|00|00|00|00|00|00|00|
2022-05-23T03:55:37Z RX-gen on port 0 queue 0
2022-05-23T03:55:37Z
2022-05-23T03:55:37Z +---------+---------------+----------+
2022-05-23T03:55:37Z 00:00:00,000,000 ETHER
2022-05-23T03:55:37Z |0 |ff|ff|ff|ff|ff|ff|e4|5d|37|0f|ef|99|08|06|00|01|08|00|06|04|00|01|e4|5d|37|0f|ef|99|0a|ff|00|03|00|00|00|00|00|00|0a|ff|06|0f|36|8f|b7|05|00|00|00|00|00|00|00|00|00|00|00|00|00|00|
2022-05-23T03:55:37Z RX ARP request on port 0 queue 0 sip:10.255.0.3 tip:10.255.6.15 vlan:0
2022-05-23T03:55:37Z
2022-05-23T03:55:37Z +---------+---------------+----------+
2022-05-23T03:55:37Z 00:00:00,000,000 ETHER
2022-05-23T03:55:37Z |0 |ff|ff|ff|ff|ff|ff|e4|5d|37|0f|ef|99|08|06|00|01|08|00|06|04|00|01|e4|5d|37|0f|ef|99|0a|ff|00|03|00|00|00|00|00|00|0a|ff|06|0f|36|8f|b7|05|00|00|00|00|00|00|00|00|00|00|00|00|00|00|
2022-05-23T03:55:37Z RX-gen on port 0 queue 0
2022-05-23T03:55:37Z
2022-05-23T03:55:37Z +---------+---------------+----------+
2022-05-23T03:55:37Z 00:00:00,000,000 ETHER
2022-05-23T03:55:37Z |0 |ff|ff|ff|ff|ff|ff|e4|5d|37|0f|ef|99|08|06|00|01|08|00|06|04|00|01|e4|5d|37|0f|ef|99|0a|ff|00|03|00|00|00|00|00|00|0a|ff|06|a6|e0|99|32|ae|00|00|00|00|00|00|00|00|00|00|00|00|00|00|
2022-05-23T03:55:37Z RX ARP request on port 0 queue 0 sip:10.255.0.3 tip:10.255.6.166 vlan:0
2022-05-23T03:55:37Z
2022-05-23T03:55:37Z +---------+---------------+----------+
2022-05-23T03:55:37Z 00:00:00,000,000 ETHER
2022-05-23T03:55:37Z |0 |ff|ff|ff|ff|ff|ff|e4|5d|37|0f|ef|99|08|06|00|01|08|00|06|04|00|01|e4|5d|37|0f|ef|99|0a|ff|00|03|00|00|00|00|00|00|0a|ff|06|a6|e0|99|32|ae|00|00|00|00|00|00|00|00|00|00|00|00|00|00|
2022-05-23T03:55:37Z RX-gen on port 0 queue 0
2022-05-23T03:55:37Z
2022-05-23T03:55:37Z +---------+---------------+----------+
2022-05-23T03:55:37Z 00:00:00,000,000 ETHER
2022-05-23T03:55:37Z |0 |ff|ff|ff|ff|ff|ff|e4|5d|37|09|bf|99|08|06|00|01|08|00|06|04|00|01|e4|5d|37|09|bf|99|0a|ff|00|02|00|00|00|00|00|00|0a|ff|06|a8|9a|99|eb|a5|00|00|00|00|00|00|00|00|00|00|00|00|00|00|
2022-05-23T03:55:37Z RX ARP request on port 0 queue 0 sip:10.255.0.2 tip:10.255.6.168 vlan:0
2022-05-23T03:55:37Z
2022-05-23T03:55:37Z +---------+---------------+----------+
2022-05-23T03:55:37Z 00:00:00,000,000 ETHER
2022-05-23T03:55:37Z |0 |ff|ff|ff|ff|ff|ff|e4|5d|37|09|bf|99|08|06|00|01|08|00|06|04|00|01|e4|5d|37|09|bf|99|0a|ff|00|02|00|00|00|00|00|00|0a|ff|06|a8|9a|99|eb|a5|00|00|00|00|00|00|00|00|00|00|00|00|00|00|
2022-05-23T03:55:37Z RX-gen on port 0 queue 0
2022-05-23T03:55:37Z
2022-05-23T03:55:37Z +---------+---------------+----------+
2022-05-23T03:55:37Z 00:00:00,000,000 ETHER
2022-05-23T03:55:37Z |0 |ff|ff|ff|ff|ff|ff|e4|5d|37|09|bf|99|08|06|00|01|08|00|06|04|00|01|e4|5d|37|09|bf|99|0a|ff|00|02|00|00|00|00|00|00|0a|ff|06|a9|5d|a3|b0|4b|00|00|00|00|00|00|00|00|00|00|00|00|00|00|
2022-05-23T03:55:37Z RX ARP request on port 0 queue 0 sip:10.255.0.2 tip:10.255.6.169 vlan:0
2022-05-23T03:55:37Z
2022-05-23T03:55:37Z +---------+---------------+----------+
2022-05-23T03:55:37Z 00:00:00,000,000 ETHER
2022-05-23T03:55:37Z |0 |ff|ff|ff|ff|ff|ff|e4|5d|37|09|bf|99|08|06|00|01|08|00|06|04|00|01|e4|5d|37|09|bf|99|0a|ff|00|02|00|00|00|00|00|00|0a|ff|06|a9|5d|a3|b0|4b|00|00|00|00|00|00|00|00|00|00|00|00|00|00|
2022-05-23T03:55:37Z RX-gen on port 0 queue 0
2022-05-23T03:55:37Z
2022-05-23T03:55:37Z +---------+---------------+----------+
2022-05-23T03:55:37Z 00:00:00,000,000 ETHER
2022-05-23T03:55:37Z |0 |ff|ff|ff|ff|ff|ff|e4|5d|37|0f|ef|99|08|06|00|01|08|00|06|04|00|01|e4|5d|37|0f|ef|99|0a|ff|00|03|00|00|00|00|00|00|0a|ff|06|0e|06|e7|1a|b5|00|00|00|00|00|00|00|00|00|00|00|00|00|00|
2022-05-23T03:55:37Z RX ARP request on port 0 queue 0 sip:10.255.0.3 tip:10.255.6.14 vlan:0
2022-05-23T03:55:37Z
2022-05-23T03:55:37Z +---------+---------------+----------+
2022-05-23T03:55:37Z 00:00:00,000,000 ETHER
2022-05-23T03:55:37Z |0 |ff|ff|ff|ff|ff|ff|e4|5d|37|0f|ef|99|08|06|00|01|08|00|06|04|00|01|e4|5d|37|0f|ef|99|0a|ff|00|03|00|00|00|00|00|00|0a|ff|06|0e|06|e7|1a|b5|00|00|00|00|00|00|00|00|00|00|00|00|00|00|
2022-05-23T03:55:37Z RX-gen on port 0 queue 0
2022-05-23T03:55:37Z
2022-05-23T03:55:37Z +---------+---------------+----------+
2022-05-23T03:55:37Z 00:00:00,000,000 ETHER
2022-05-23T03:55:37Z |0 |ff|ff|ff|ff|ff|ff|e4|5d|37|0f|ef|99|08|06|00|01|08|00|06|04|00|01|e4|5d|37|0f|ef|99|0a|ff|00|03|00|00|00|00|00|00|0a|ff|06|a9|f1|d5|5d|b0|00|00|00|00|00|00|00|00|00|00|00|00|00|00|
2022-05-23T03:55:37Z RX ARP request on port 0 queue 0 sip:10.255.0.3 tip:10.255.6.169 vlan:0
2022-05-23T03:55:37Z
2022-05-23T03:55:37Z +---------+---------------+----------+
2022-05-23T03:55:37Z 00:00:00,000,000 ETHER
2022-05-23T03:55:37Z |0 |ff|ff|ff|ff|ff|ff|e4|5d|37|0f|ef|99|08|06|00|01|08|00|06|04|00|01|e4|5d|37|0f|ef|99|0a|ff|00|03|00|00|00|00|00|00|0a|ff|06|a9|f1|d5|5d|b0|00|00|00|00|00|00|00|00|00|00|00|00|00|00|
2022-05-23T03:55:37Z RX-gen on port 0 queue 0
2022-05-23T03:55:37Z
2022-05-23T03:55:37Z +---------+---------------+----------+
2022-05-23T03:55:37Z 00:00:00,000,000 ETHER
2022-05-23T03:55:37Z |0 |ff|ff|ff|ff|ff|ff|e4|5d|37|0f|ef|99|08|06|00|01|08|00|06|04|00|01|e4|5d|37|0f|ef|99|0a|ff|00|03|00|00|00|00|00|00|0a|ff|07|e6|05|ab|a9|d5|00|00|00|00|00|00|00|00|00|00|00|00|00|00|
2022-05-23T03:55:37Z RX ARP request on port 0 queue 0 sip:10.255.0.3 tip:10.255.7.230 vlan:0
2022-05-23T03:55:37Z
2022-05-23T03:55:37Z +---------+---------------+----------+
2022-05-23T03:55:37Z 00:00:00,000,000 ETHER
2022-05-23T03:55:37Z |0 |ff|ff|ff|ff|ff|ff|e4|5d|37|0f|ef|99|08|06|00|01|08|00|06|04|00|01|e4|5d|37|0f|ef|99|0a|ff|00|03|00|00|00|00|00|00|0a|ff|07|e6|05|ab|a9|d5|00|00|00|00|00|00|00|00|00|00|00|00|00|00|
2022-05-23T03:55:37Z RX-gen on port 0 queue 0
2022-05-23T03:55:37Z
2022-05-23T03:55:37Z +---------+---------------+----------+
2022-05-23T03:55:37Z 00:00:00,000,000 ETHER
2022-05-23T03:55:37Z |0 |ff|ff|ff|ff|ff|ff|e4|43|4b|9a|c2|a8|08|00|45|00|00|3c|eb|95|40|00|40|11|2f|2e|0a|ff|01|f1|0a|ff|07|ff|89|7f|17|01|00|28|0e|80|08|c8|01|c2|0c|1a|0d|f1|01|ff|0a|10|d7|5e|1a|10|73|65|72|6f|69|61|75|65|73|69|6d|30|30|33|39|34|
2022-05-23T03:55:37Z RX-gen on port 0 queue 0
2022-05-23T03:55:37Z
2022-05-23T03:55:37Z +---------+---------------+----------+
2022-05-23T03:55:37Z 00:00:00,000,000 ETHER
2022-05-23T03:55:37Z |0 |ff|ff|ff|ff|ff|ff|e4|5d|37|09|bf|99|08|06|00|01|08|00|06|04|00|01|e4|5d|37|09|bf|99|0a|ff|00|02|00|00|00|00|00|00|0a|ff|06|8c|cd|45|0d|66|00|00|00|00|00|00|00|00|00|00|00|00|00|00|
2022-05-23T03:55:37Z RX ARP request on port 0 queue 0 sip:10.255.0.2 tip:10.255.6.140 vlan:0
2022-05-23T03:55:37Z
2022-05-23T03:55:37Z +---------+---------------+----------+
2022-05-23T03:55:37Z 00:00:00,000,000 ETHER
2022-05-23T03:55:37Z |0 |ff|ff|ff|ff|ff|ff|e4|5d|37|09|bf|99|08|06|00|01|08|00|06|04|00|01|e4|5d|37|09|bf|99|0a|ff|00|02|00|00|00|00|00|00|0a|ff|06|8c|cd|45|0d|66|00|00|00|00|00|00|00|00|00|00|00|00|00|00|
2022-05-23T03:55:37Z RX-gen on port 0 queue 0
2022-05-23T03:55:37Z
2022-05-23T03:55:37Z +---------+---------------+----------+
2022-05-23T03:55:37Z 00:00:00,000,000 ETHER
2022-05-23T03:55:37Z |0 |ff|ff|ff|ff|ff|ff|e4|5d|37|0f|ef|99|08|06|00|01|08|00|06|04|00|01|e4|5d|37|0f|ef|99|0a|ff|00|03|00|00|00|00|00|00|0a|ff|06|41|5d|0c|c0|bf|00|00|00|00|00|00|00|00|00|00|00|00|00|00|
2022-05-23T03:55:37Z RX ARP request on port 0 queue 0 sip:10.255.0.3 tip:10.255.6.65 vlan:0
2022-05-23T03:55:37Z
2022-05-23T03:55:37Z +---------+---------------+----------+
2022-05-23T03:55:37Z 00:00:00,000,000 ETHER
2022-05-23T03:55:37Z |0 |ff|ff|ff|ff|ff|ff|e4|5d|37|0f|ef|99|08|06|00|01|08|00|06|04|00|01|e4|5d|37|0f|ef|99|0a|ff|00|03|00|00|00|00|00|00|0a|ff|06|41|5d|0c|c0|bf|00|00|00|00|00|00|00|00|00|00|00|00|00|00|
2022-05-23T03:55:37Z RX-gen on port 0 queue 0
2022-05-23T03:55:37Z
2022-05-23T03:55:37Z +---------+---------------+----------+
2022-05-23T03:55:37Z 00:00:00,000,000 ETHER
2022-05-23T03:55:37Z |0 |ff|ff|ff|ff|ff|ff|e4|5d|37|0f|ef|99|08|06|00|01|08|00|06|04|00|01|e4|5d|37|0f|ef|99|0a|ff|00|03|00|00|00|00|00|00|0a|ff|06|ae|82|17|73|b2|00|00|00|00|00|00|00|00|00|00|00|00|00|00|
2022-05-23T03:55:37Z RX ARP request on port 0 queue 0 sip:10.255.0.3 tip:10.255.6.174 vlan:0
2022-05-23T03:55:37Z
2022-05-23T03:55:37Z +---------+---------------+----------+
2022-05-23T03:55:37Z 00:00:00,000,000 ETHER
2022-05-23T03:55:37Z |0 |ff|ff|ff|ff|ff|ff|e4|5d|37|0f|ef|99|08|06|00|01|08|00|06|04|00|01|e4|5d|37|0f|ef|99|0a|ff|00|03|00|00|00|00|00|00|0a|ff|06|ae|82|17|73|b2|00|00|00|00|00|00|00|00|00|00|00|00|00|00|
2022-05-23T03:55:37Z RX-gen on port 0 queue 0
2022-05-23T03:55:37Z
2022-05-23T03:55:37Z +---------+---------------+----------+
2022-05-23T03:55:37Z 00:00:00,000,000 ETHER
2022-05-23T03:55:37Z |0 |ff|ff|ff|ff|ff|ff|e4|43|4b|65|a2|64|08|00|45|00|00|3c|52|ea|40|00|40|11|c9|b6|0a|ff|00|14|0a|ff|07|ff|92|c8|17|01|00|28|09|ee|08|c8|01|c2|0c|1a|0d|14|00|ff|0a|10|d7|5e|1a|10|73|65|72|6f|69|61|75|65|73|69|6d|30|30|32|38|38|
2022-05-23T03:55:37Z TX ARP request on port 1 - ip: 10.255.6.133 mac: Unknown
2022-05-23T03:55:37Z
2022-05-23T03:55:37Z +---------+---------------+----------+
2022-05-23T03:55:37Z 00:00:00,000,000 ETHER
2022-05-23T03:55:37Z |0 |ff|ff|ff|ff|ff|ff|2e|8e|09|fe|1f|0b|08|06|00|01|08|00|06|04|00|01|2e|8e|09|fe|1f|0b|0a|ff|06|84|01|03|05|07|09|01|0a|ff|06|85|00|00|00|00|00|00|00|00|00|00|00|00|00|00|00|00|00|00|
2022-05-23T03:55:37Z RX-gen on port 0 queue 0
2022-05-23T03:55:37Z
2022-05-23T03:55:37Z +---------+---------------+----------+
2022-05-23T03:55:37Z 00:00:00,000,000 ETHER
2022-05-23T03:55:37Z |0 |ff|ff|ff|ff|ff|ff|2e|8e|09|fe|1f|0b|08|06|00|01|08|00|06|04|00|01|2e|8e|09|fe|1f|0b|0a|ff|06|84|01|03|05|07|09|01|0a|ff|06|85|00|00|00|00|00|00|00|00|00|00|00|00|00|00|00|00|00|00|
2022-05-23T03:55:37Z RX ARP request on port 0 queue 0 sip:10.255.6.132 tip:10.255.6.133 vlan:0
2022-05-23T03:55:37Z
2022-05-23T03:55:37Z +---------+---------------+----------+
2022-05-23T03:55:37Z 00:00:00,000,000 ETHER
2022-05-23T03:55:37Z |0 |ff|ff|ff|ff|ff|ff|2e|8e|09|fe|1f|0b|08|06|00|01|08|00|06|04|00|01|2e|8e|09|fe|1f|0b|0a|ff|06|84|01|03|05|07|09|01|0a|ff|06|85|00|00|00|00|00|00|00|00|00|00|00|00|00|00|00|00|00|00|
2022-05-23T03:55:37Z RX-gen on port 0 queue 0
2022-05-23T03:55:37Z

Answer 1 · 2022-05-29T10:06:51.000Z

@egwakim the driver was changed to iavf driver
The old driver was removed (i40evf )
One workaround is to add the old driver back and accommodate the new API (shouldn't be a big issue) and report this issue in DPDK forum

Answer 2 · 2022-05-30T00:49:03.000Z

It's issue on DPDK 22.03 iavf PMD,
Proper way would be fixing the issue from DPDK.
I reported it to DPDK bugzilla.
https://bugs.dpdk.org/show_bug.cgi?id=1019

Answer 3 · 2022-05-30T13:13:57.000Z

@egwakim I don't see a patch or a workaround yet

Answer 4 · 2022-06-07T03:32:54.000Z

@hhaim, according to my examination, in this case, the current RX descriptor (indexed by rxq->rx_tail) is not done (status_error_len = 0), but the next descriptor has been done (status_error_len = 16504485606403).

(gdb) p *(rxq->rx_ring + 0)
$24 = {read = {pkt_addr = 7316192320, hdr_addr = 0, rsvd1 = 0, rsvd2 = 0}, wb = {qword0 = {lo_dword = {
        mirr_fcoe = {mirroring_status = 15424, fcoe_ctx_id = 15424}, l2tag1 = 46100}, hi_dword = {rss = 1, 
        fcoe_param = 1, fd_id = 1}}, qword1 = {status_error_len = 0}, qword2 = {ext_status = 0, rsvd = 0, 
      l2tag2_1 = 0, l2tag2_2 = 0}, qword3 = {lo_dword = {flex_bytes_lo = 0, pe_status = 0}, hi_dword = {
        flex_bytes_hi = 0, fd_id = 0}}}}
(gdb) p *(rxq->rx_ring + 1)
$25 = {read = {pkt_addr = 16595853889165590528, hdr_addr = 16504485606403, rsvd1 = 0, rsvd2 = 0}, wb = {qword0 = {
      lo_dword = {mirr_fcoe = {mirroring_status = 0, fcoe_ctx_id = 0}, l2tag1 = 0}, hi_dword = {rss = 3864023343, 
        fcoe_param = 3864023343, fd_id = 3864023343}}, qword1 = {status_error_len = 16504485606403}, qword2 = {
      ext_status = 0, rsvd = 0, l2tag2_1 = 0, l2tag2_2 = 0}, qword3 = {lo_dword = {flex_bytes_lo = 0, 
        pe_status = 0}, hi_dword = {flex_bytes_hi = 0, fd_id = 0}}}}

When I move the current index to the next descriptor, the delivered packets can be received through the rx_burst API.

diff --git a/src/dpdk/drivers/net/iavf/iavf_rxtx_vec_sse.c b/src/dpdk/drivers/net/iavf/iavf_rxtx_vec_sse.c
index 717a227b2..09a817b53 100644
--- a/src/dpdk/drivers/net/iavf/iavf_rxtx_vec_sse.c
+++ b/src/dpdk/drivers/net/iavf/iavf_rxtx_vec_sse.c
@@ -437,8 +439,19 @@ _recv_raw_pkts_vec(struct iavf_rx_queue *rxq, struct rte_mbuf **rx_pkts,
         * there is actually a packet available
         */
        if (!(rxdp->wb.qword1.status_error_len &
-             rte_cpu_to_le_32(1 << IAVF_RX_DESC_STATUS_DD_SHIFT)))
+             rte_cpu_to_le_32(1 << IAVF_RX_DESC_STATUS_DD_SHIFT))) {
+                uint16_t rx_tail = (uint16_t)((rxq->rx_tail + 1) & (rxq->nb_rx_desc - 1));
+                rxdp = rxq->rx_ring + rx_tail;
+                if ((rxdp->wb.qword1.status_error_len &
+                      rte_cpu_to_le_32(1 << IAVF_RX_DESC_STATUS_DD_SHIFT))) {
+                        PMD_DRV_LOG(WARNING, "port_id=%u queue_id=%u rx_tail=%u is skipped "
+                                   "because next packet is available",
+                                   rxq->port_id, rxq->queue_id, rxq->rx_tail);
+                        rxq->rx_tail = rx_tail;
+                }
                return 0;
+        }
 
        /* 4 packets DD mask */
        dd_check = _mm_set_epi64x(0x0000000100000001LL, 0x0000000100000001LL);

I think this is a simple workaround that can be used temporarily before the root cause is identified and fixed.
What do you think about this?

Answer 5 · 2022-06-07T06:16:37.000Z

@jsmoon the issue was already workaround in master. we took the old i40evf dpdk driver and accommodate it to the new API (minor)
In next DPDK version we will revisit this driver (which is the future)

Answer 6 · 2022-06-07T07:17:29.000Z

@hhaim thank you for your work.

Answer 7 · 2022-06-07T07:39:12.000Z

@jsmoon just released. @syaakov looked into this and fixed the issue

Answer 8 · 2022-06-07T09:33:21.000Z

@syaakov thank you so much. I've tried the patchwork from the DPDK commit but failed to make it work.

Answer 9 · 2022-09-19T02:58:59.000Z

@hhaim @syaakov my colleague wants to use the IAVF driver for the E810 device in T-Rex.
Since there is no progress on the DPDK side, I'm not sure that the next DPDK version can resolve it.
So I think a way to enable it temporarily until then.
Since the IAVF device id 0x1889 is distinguished from i40e VF device IDs, I think I can make it activated for the IAVF driver only.
Please check my change at here and give your opinion.

Answer 10 · 2022-09-19T11:31:47.000Z

@jsmoon looks good, please PR it

Answer 11 · 2023-10-05T06:22:39.000Z

Hi @hhaim
This issue is re-happening from DPDK 23.03.

Issue
All iavf ports hang if broadcast packets received during TRex initialization.
Reproduction : 100% reproducible if broadcast packets flooded

Run flood ping with broadcast in same subnet

sudo ping -f -b -q 1.1.1.255 -I enp59s13f3 # enp59s13f3 is one of IAVF which shares same PF with TRex ports

It's happening whatever the latest version of iavf kernel driver used or i40evf driver used.

Start TRex
Failed to get MAC, all packets are dropped and marked as ierror and rx_missed error
trex>stats -x
Xstats
```
     Name:              |     Port 0:     |     Port 1:     
```

-------------------------------+-----------------+----------------
rx_good_packets | 522 | 521
tx_good_packets | 11 | 11
rx_good_bytes | 84596 | 79896
tx_good_bytes | 660 | 660
rx_missed_errors | 373 | 326
rx_bytes | 86684 | 81980
rx_broadcast_packets | 895 | 847
rx_dropped_packets | 373 | 326
tx_bytes | 660 | 660
tx_broadcast_packets | 11 | 11

Found kernel driver error
kernel: DMAR: [DMA Write] Request device [3b:0d.1] fault addr 0 [fault reason 05] PTE Write access is not set

It happens when packets received during iavf_dev_rx_queue_start() call.