Number of memzone segments exceeds RTE_MAX_MEMZONE
beyonson opened this issue · 5 comments
beyonson commented
Hello, I am trying to run t-rex-64 using 4 threads per interface pair (-c flag = 4). It works with 3 threads per interface. However, whenever I try to run with 4, I get the following:
$ ./t-rex-64 -f cap2/imix_64_e810_max.yaml --cfg cfg/e810c.yaml -c 4
...
EAL: memzone_reserve_aligned_thread_unsafe(): Number of requested memzone segments exceeds RTE_MAX_MEMZONE
RING: Cannot reserve memory
ERROR there is not enough huge-pages memory in your system
EAL: Error - exiting with code: 1
Cause: Cannot init nodes mbuf pool nodes-27
I have tried increasing the number of hugepages to a pretty large size, to no avail. In the previous run I had 64GB of 1GB hugepages. I have tried up to 200GB. I have tried this with 2048kB hugepages and 1GB hugepages.
$ cat /proc/meminfo | grep Huge
AnonHugePages: 0 kB
ShmemHugePages: 0 kB
FileHugePages: 0 kB
HugePages_Total: 64
HugePages_Free: 61
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 1048576 kB
Hugetlb: 71303168 kB
Here is my config file:
- port_limit : 16
version : 2
#List of interfaces. Change to suit your setup. Use ./dpdk_setup_ports.py -s to see available options
interfaces : ["01:00.0","01:00.1","01:00.2","01:00.3","01:00.4","01:00.5","01:00.6","01:00.7","61:00.0","61:00.1
","61:00.2","61:00.3","61:00.4","61:00.5","61:00.6","61:00.7"]
port_info : # Port IPs. Change to suit your needs. In case of loopback, you can leave as is.
- ip : 1.1.1.1 # port 0
default_gw : 2.2.2.2
- ip : 2.2.2.2 # port 1
default_gw : 1.1.1.1
...
- ip : 88.88.88.88 # port 15
default_gw : 77.77.77.77
Here is my traffic file:
#
# Simple IMIX test 64B
#
- duration : 3
generator :
distribution : "seq"
clients_start : "16.0.0.1"
clients_end : "16.0.0.255"
servers_start : "48.0.0.1"
servers_end : "48.0.255.255"
dual_port_mask : "1.0.0.0"
tcp_aging : 0
udp_aging : 0
#
# The templates are duplicated on purpose in order to utilize all DRAM bandwidth and get better performance
# Should do this automaticlly in the future somehow. You should have at least 8 templates
cap_info :
- name: cap2/udp_64B_bidirectional.pcap
cps : 1000.0
ipg : 0
rtt : 0
w : 1
...
# I am loading 8 pcaps.
- name: cap2/udp_64B_bidirectional.pcap
ipg : 0
rtt : 0
w : 1
Lastly, here is a verbose output of my run:
$ ./t-rex-64 -f cap2/imix_64_e810_max.yaml --cfg cfg/e810c.yaml -c 4 -v 7
The ports are bound/configured.
Starting TRex v3.03 please wait ...
Using configuration file cfg/e810c.yaml
port limit : 16
port_bandwidth_gb : 10
port_speed : 0
port_mtu : 0
if_mask : None
is low-end : 0
stack type :
thread_per_dual_if : 1
if : 01:00.0, 01:00.1, 01:00.2, 01:00.3, 01:00.4, 01:00.5, 01:00.6, 01:00.7, 61:00.0, 61:00.1, 61:00.2, 61:00.3, 61:00.4, 61:00.5, 61:00.6, 61:00.7,
enable_zmq_pub : 1
zmq_pub_port : 4500
m_zmq_rpc_port : 4501
src : 00:00:00:00:00:00
dest : 00:00:00:00:00:00
src : 00:00:00:00:00:00
dest : 00:00:00:00:00:00
src : 00:00:00:00:00:00
dest : 00:00:00:00:00:00
src : 00:00:00:00:00:00
dest : 00:00:00:00:00:00
src : 00:00:00:00:00:00
dest : 00:00:00:00:00:00
src : 00:00:00:00:00:00
dest : 00:00:00:00:00:00
src : 00:00:00:00:00:00
dest : 00:00:00:00:00:00
src : 00:00:00:00:00:00
dest : 00:00:00:00:00:00
src : 00:00:00:00:00:00
dest : 00:00:00:00:00:00
src : 00:00:00:00:00:00
dest : 00:00:00:00:00:00
src : 00:00:00:00:00:00
dest : 00:00:00:00:00:00
src : 00:00:00:00:00:00
dest : 00:00:00:00:00:00
src : 00:00:00:00:00:00
dest : 00:00:00:00:00:00
src : 00:00:00:00:00:00
dest : 00:00:00:00:00:00
src : 00:00:00:00:00:00
dest : 00:00:00:00:00:00
src : 00:00:00:00:00:00
dest : 00:00:00:00:00:00
memory per 2x10G ports
MBUF_64 : 16380
MBUF_128 : 8190
MBUF_256 : 8190
MBUF_512 : 8190
MBUF_1024 : 8190
MBUF_2048 : 4095
MBUF_4096 : 128
MBUF_9K : 512
TRAFFIC_MBUF_64 : 65520
TRAFFIC_MBUF_128 : 32760
TRAFFIC_MBUF_256 : 8190
TRAFFIC_MBUF_512 : 8190
TRAFFIC_MBUF_1024 : 8190
TRAFFIC_MBUF_2048 : 32760
TRAFFIC_MBUF_4096 : 128
TRAFFIC_MBUF_9K : 512
MBUF_DP_FLOWS : 524288
MBUF_GLOBAL_FLOWS : 5120
no platform info
CTimerWheelYamlInfo does not exist
flags : 8040f00
write_file : 0
verbose : 7
realtime : 1
flip : 0
cores : 4
single core : 0
flow-flip : 0
no clean close : 0
zmq_publish : 1
vlan mode : 0
client_cfg : 0
mbuf_cache_disable : 0
cfg file : cap2/imix_64_e810_max.yaml
mac file :
out file :
client cfg file :
duration : 3600
factor : 1
mbuf_factor : 1
latency : 0 pkt/sec
zmq_port : 4500
telnet_port : 4501
expected_ports : 16
tw_bucket_usec : 20.000000 usec
tw_buckets : 1024 usec
tw_levels : 3 usec
port : 0 dst:00:00:00:00:00:00 src:00:00:00:00:00:00
port : 1 dst:00:00:00:00:00:00 src:00:00:00:00:00:00
port : 2 dst:00:00:00:00:00:00 src:00:00:00:00:00:00
port : 3 dst:00:00:00:00:00:00 src:00:00:00:00:00:00
port : 4 dst:00:00:00:00:00:00 src:00:00:00:00:00:00
port : 5 dst:00:00:00:00:00:00 src:00:00:00:00:00:00
port : 6 dst:00:00:00:00:00:00 src:00:00:00:00:00:00
port : 7 dst:00:00:00:00:00:00 src:00:00:00:00:00:00
port : 8 dst:00:00:00:00:00:00 src:00:00:00:00:00:00
port : 9 dst:00:00:00:00:00:00 src:00:00:00:00:00:00
port : 10 dst:00:00:00:00:00:00 src:00:00:00:00:00:00
port : 11 dst:00:00:00:00:00:00 src:00:00:00:00:00:00
port : 12 dst:00:00:00:00:00:00 src:00:00:00:00:00:00
port : 13 dst:00:00:00:00:00:00 src:00:00:00:00:00:00
port : 14 dst:00:00:00:00:00:00 src:00:00:00:00:00:00
port : 15 dst:00:00:00:00:00:00 src:00:00:00:00:00:00
port : 16 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 17 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 18 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 19 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 20 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 21 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 22 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 23 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 24 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 25 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 26 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 27 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 28 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 29 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 30 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 31 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
Total Memory :
MBUF_64 : 196560
MBUF_128 : 98280
MBUF_256 : 73710
MBUF_512 : 73710
MBUF_1024 : 73710
MBUF_2048 : 65520
MBUF_4096 : 32768
MBUF_DP_FLOWS : 4194304
MBUF_GLOBAL_FLOWS : 40960
get_each_core_dp_flows : 131072
Total memory : 1.26 Gbytes
there is no configuration file given
DPDK args
xx -l 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33 -n 4 --log-level 8 --main-lcore 0 -a 0000:01:00.0 -a 0000:01:00.1 -a 0000:01:00.2 -a 0000:01:00.3 -a 0000:01:00.4 -a 0000:01:00.5 -a 0000:01:00.6 -a 0000:01:00.7 -a 0000:61:00.0 -a 0000:61:00.1 -a 0000:61:00.2 -a 0000:61:00.3 -a 0000:61:00.4 -a 0000:61:00.5 -a 0000:61:00.6 -a 0000:61:00.7
EAL: Detected CPU lcores: 64
EAL: Detected NUMA nodes: 1
EAL: Detected static linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'VA'
EAL: 2048 hugepages of size 2097152 reserved, but no mounted hugetlbfs found for that size
EAL: VFIO support initialized
EAL: Using IOMMU type 1 (Type 1)
EAL: Probe PCI driver: net_ice (8086:1592) device: 0000:01:00.0 (socket 0)
ice_load_pkg_type(): Active package is: 1.3.30.0, ICE OS Default Package (single VLAN mode)
EAL: Probe PCI driver: net_ice (8086:1592) device: 0000:01:00.1 (socket 0)
ice_load_pkg_type(): Active package is: 1.3.30.0, ICE OS Default Package (single VLAN mode)
EAL: Probe PCI driver: net_ice (8086:1592) device: 0000:01:00.2 (socket 0)
ice_load_pkg_type(): Active package is: 1.3.30.0, ICE OS Default Package (single VLAN mode)
EAL: Probe PCI driver: net_ice (8086:1592) device: 0000:01:00.3 (socket 0)
ice_load_pkg_type(): Active package is: 1.3.30.0, ICE OS Default Package (single VLAN mode)
EAL: Probe PCI driver: net_ice (8086:1592) device: 0000:01:00.4 (socket 0)
ice_load_pkg_type(): Active package is: 1.3.30.0, ICE OS Default Package (single VLAN mode)
EAL: Probe PCI driver: net_ice (8086:1592) device: 0000:01:00.5 (socket 0)
ice_load_pkg_type(): Active package is: 1.3.30.0, ICE OS Default Package (single VLAN mode)
EAL: Probe PCI driver: net_ice (8086:1592) device: 0000:01:00.6 (socket 0)
ice_load_pkg_type(): Active package is: 1.3.30.0, ICE OS Default Package (single VLAN mode)
EAL: Probe PCI driver: net_ice (8086:1592) device: 0000:01:00.7 (socket 0)
ice_load_pkg_type(): Active package is: 1.3.30.0, ICE OS Default Package (single VLAN mode)
EAL: Probe PCI driver: net_ice (8086:1592) device: 0000:61:00.0 (socket 0)
ice_load_pkg_type(): Active package is: 1.3.30.0, ICE OS Default Package (single VLAN mode)
EAL: Probe PCI driver: net_ice (8086:1592) device: 0000:61:00.1 (socket 0)
ice_load_pkg_type(): Active package is: 1.3.30.0, ICE OS Default Package (single VLAN mode)
EAL: Probe PCI driver: net_ice (8086:1592) device: 0000:61:00.2 (socket 0)
ice_load_pkg_type(): Active package is: 1.3.30.0, ICE OS Default Package (single VLAN mode)
EAL: Probe PCI driver: net_ice (8086:1592) device: 0000:61:00.3 (socket 0)
ice_load_pkg_type(): Active package is: 1.3.30.0, ICE OS Default Package (single VLAN mode)
EAL: Probe PCI driver: net_ice (8086:1592) device: 0000:61:00.4 (socket 0)
ice_load_pkg_type(): Active package is: 1.3.30.0, ICE OS Default Package (single VLAN mode)
EAL: Probe PCI driver: net_ice (8086:1592) device: 0000:61:00.5 (socket 0)
ice_load_pkg_type(): Active package is: 1.3.30.0, ICE OS Default Package (single VLAN mode)
EAL: Probe PCI driver: net_ice (8086:1592) device: 0000:61:00.6 (socket 0)
ice_load_pkg_type(): Active package is: 1.3.30.0, ICE OS Default Package (single VLAN mode)
EAL: Probe PCI driver: net_ice (8086:1592) device: 0000:61:00.7 (socket 0)
ice_load_pkg_type(): Active package is: 1.3.30.0, ICE OS Default Package (single VLAN mode)
TELEMETRY: No legacy callbacks, legacy socket not created
input : [01:00.0, 01:00.1, 01:00.2, 01:00.3, 01:00.4, 01:00.5, 01:00.6, 01:00.7, 61:00.0, 61:00.1, 61:00.2, 61:00.3, 61:00.4, 61:00.5, 61:00.6, 61:00.7]
dpdk : [0000:01:00.0, 0000:01:00.1, 0000:01:00.2, 0000:01:00.3, 0000:01:00.4, 0000:01:00.5, 0000:01:00.6, 0000:01:00.7, 0000:61:00.0, 0000:61:00.1, 0000:61:00.2, 0000:61:00.3, 0000:61:00.4, 0000:61:00.5, 0000:61:00.6, 0000:61:00.7]
pci_scan : [0000:01:00.0, 0000:01:00.1, 0000:01:00.2, 0000:01:00.3, 0000:01:00.4, 0000:01:00.5, 0000:01:00.6, 0000:01:00.7, 0000:61:00.0, 0000:61:00.1, 0000:61:00.2, 0000:61:00.3, 0000:61:00.4, 0000:61:00.5, 0000:61:00.6, 0000:61:00.7]
map : [ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]
TRex port mapping
-----------------
TRex vport: 0 dpdk_rte_eth: 0
TRex vport: 1 dpdk_rte_eth: 1
TRex vport: 2 dpdk_rte_eth: 2
TRex vport: 3 dpdk_rte_eth: 3
TRex vport: 4 dpdk_rte_eth: 4
TRex vport: 5 dpdk_rte_eth: 5
TRex vport: 6 dpdk_rte_eth: 6
TRex vport: 7 dpdk_rte_eth: 7
TRex vport: 8 dpdk_rte_eth: 8
TRex vport: 9 dpdk_rte_eth: 9
TRex vport: 10 dpdk_rte_eth: 10
TRex vport: 11 dpdk_rte_eth: 11
TRex vport: 12 dpdk_rte_eth: 12
TRex vport: 13 dpdk_rte_eth: 13
TRex vport: 14 dpdk_rte_eth: 14
TRex vport: 15 dpdk_rte_eth: 15
set driver name net_ice
driver capability : TCP_UDP_OFFLOAD TSO SLRO
set dpdk queues mode to DROP_QUE_FILTER
DPDK devices 16 : 16
-----
0 : vdev 0000:01:00.0
1 : vdev 0000:01:00.1
2 : vdev 0000:01:00.2
3 : vdev 0000:01:00.3
4 : vdev 0000:01:00.4
5 : vdev 0000:01:00.5
6 : vdev 0000:01:00.6
7 : vdev 0000:01:00.7
8 : vdev 0000:61:00.0
9 : vdev 0000:61:00.1
10 : vdev 0000:61:00.2
11 : vdev 0000:61:00.3
12 : vdev 0000:61:00.4
13 : vdev 0000:61:00.5
14 : vdev 0000:61:00.6
15 : vdev 0000:61:00.7
-----
Number of ports found: 16
if_index : 0
driver name : net_ice
min_rx_bufsize : 1024
max_rx_pktlen : 9728
max_rx_queues : 64
max_tx_queues : 64
max_mac_addrs : 64
rx_offload_capa : 0x9666f
tx_offload_capa : 0x1181bf
rss reta_size : 512
flow_type_rss : 0x7ffc
tx_desc_max : 4096
tx_desc_min : 64
rx_desc_max : 4096
rx_desc_min : 64
zmq publisher at: tcp://*:4500
rx_data_q_num : 1
rx_drop_q_num : 1
rx_dp_q_num : 0
rx_que_total : 2
--
rx_desc_num_data_q : 4096
rx_desc_num_drop_q : 64
rx_desc_num_dp_q : 4096
total_desc : 4160
--
tx_desc_num : 1024
port 0 desc: Ethernet Controller E810-C for QSFP
rx_qid: 0 (64)
rx_qid: 1 (4096)
ice_set_rx_function(): Using AVX2 Vector Scattered Rx (port 0).
port 1 desc: Ethernet Controller E810-C for QSFP
rx_qid: 0 (64)
rx_qid: 1 (4096)
ice_set_rx_function(): Using AVX2 Vector Scattered Rx (port 1).
port 2 desc: Ethernet Controller E810-C for QSFP
rx_qid: 0 (64)
rx_qid: 1 (4096)
ice_set_rx_function(): Using AVX2 Vector Scattered Rx (port 2).
port 3 desc: Ethernet Controller E810-C for QSFP
rx_qid: 0 (64)
rx_qid: 1 (4096)
ice_set_rx_function(): Using AVX2 Vector Scattered Rx (port 3).
port 4 desc: Ethernet Controller E810-C for QSFP
rx_qid: 0 (64)
rx_qid: 1 (4096)
ice_set_rx_function(): Using AVX2 Vector Scattered Rx (port 4).
port 5 desc: Ethernet Controller E810-C for QSFP
rx_qid: 0 (64)
rx_qid: 1 (4096)
ice_set_rx_function(): Using AVX2 Vector Scattered Rx (port 5).
port 6 desc: Ethernet Controller E810-C for QSFP
rx_qid: 0 (64)
rx_qid: 1 (4096)
ice_set_rx_function(): Using AVX2 Vector Scattered Rx (port 6).
port 7 desc: Ethernet Controller E810-C for QSFP
rx_qid: 0 (64)
rx_qid: 1 (4096)
ice_set_rx_function(): Using AVX2 Vector Scattered Rx (port 7).
port 8 desc: Ethernet Controller E810-C for QSFP
rx_qid: 0 (64)
rx_qid: 1 (4096)
ice_set_rx_function(): Using AVX2 Vector Scattered Rx (port 8).
port 9 desc: Ethernet Controller E810-C for QSFP
rx_qid: 0 (64)
rx_qid: 1 (4096)
ice_set_rx_function(): Using AVX2 Vector Scattered Rx (port 9).
port 10 desc: Ethernet Controller E810-C for QSFP
rx_qid: 0 (64)
rx_qid: 1 (4096)
ice_set_rx_function(): Using AVX2 Vector Scattered Rx (port 10).
port 11 desc: Ethernet Controller E810-C for QSFP
rx_qid: 0 (64)
rx_qid: 1 (4096)
ice_set_rx_function(): Using AVX2 Vector Scattered Rx (port 11).
port 12 desc: Ethernet Controller E810-C for QSFP
rx_qid: 0 (64)
rx_qid: 1 (4096)
ice_set_rx_function(): Using AVX2 Vector Scattered Rx (port 12).
port 13 desc: Ethernet Controller E810-C for QSFP
rx_qid: 0 (64)
rx_qid: 1 (4096)
ice_set_rx_function(): Using AVX2 Vector Scattered Rx (port 13).
port 14 desc: Ethernet Controller E810-C for QSFP
rx_qid: 0 (64)
rx_qid: 1 (4096)
ice_set_rx_function(): Using AVX2 Vector Scattered Rx (port 14).
port 15 desc: Ethernet Controller E810-C for QSFP
rx_qid: 0 (64)
rx_qid: 1 (4096)
ice_set_rx_function(): Using AVX2 Vector Scattered Rx (port 15).
wait 1 sec .
port : 0
------------
link : link : Link Up - speed 10000 Mbps - full-duplex
promiscuous : 0
port : 1
------------
link : link : Link Up - speed 10000 Mbps - full-duplex
promiscuous : 0
port : 2
------------
link : link : Link Up - speed 10000 Mbps - full-duplex
promiscuous : 0
port : 3
------------
link : link : Link Up - speed 10000 Mbps - full-duplex
promiscuous : 0
port : 4
------------
link : link : Link Up - speed 10000 Mbps - full-duplex
promiscuous : 0
port : 5
------------
link : link : Link Up - speed 10000 Mbps - full-duplex
promiscuous : 0
port : 6
------------
link : link : Link Up - speed 10000 Mbps - full-duplex
promiscuous : 0
port : 7
------------
link : link : Link Up - speed 10000 Mbps - full-duplex
promiscuous : 0
port : 8
------------
link : link : Link Up - speed 10000 Mbps - full-duplex
promiscuous : 0
port : 9
------------
link : link : Link Up - speed 10000 Mbps - full-duplex
promiscuous : 0
port : 10
------------
link : link : Link Up - speed 10000 Mbps - full-duplex
promiscuous : 0
port : 11
------------
link : link : Link Up - speed 10000 Mbps - full-duplex
promiscuous : 0
port : 12
------------
link : link : Link Up - speed 10000 Mbps - full-duplex
promiscuous : 0
port : 13
------------
link : link : Link Up - speed 10000 Mbps - full-duplex
promiscuous : 0
port : 14
------------
link : link : Link Up - speed 10000 Mbps - full-duplex
promiscuous : 0
port : 15
------------
link : link : Link Up - speed 10000 Mbps - full-duplex
promiscuous : 0
number of ports : 16
max cores for 2 ports : 4
tx queues per port : 6
no client generator pool configured, using default pool
no server generator pool configured, using default pool
flags : 8040f00
write_file : 0
verbose : 7
realtime : 1
flip : 0
cores : 4
single core : 0
flow-flip : 0
no clean close : 0
zmq_publish : 1
vlan mode : 0
client_cfg : 0
mbuf_cache_disable : 0
cfg file : cap2/imix_64_e810_max.yaml
mac file :
out file :
client cfg file :
duration : 3600
factor : 1
mbuf_factor : 1
latency : 0 pkt/sec
zmq_port : 4500
telnet_port : 4501
expected_ports : 16
tw_bucket_usec : 20.000000 usec
tw_buckets : 1024 usec
tw_levels : 3 usec
port : 0 dst:00:00:00:00:00:00 src:64:9d:99:ff:f2:be
port : 1 dst:00:00:00:00:00:00 src:64:9d:99:ff:f2:bf
port : 2 dst:00:00:00:00:00:00 src:d0:9f:d9:70:00:e4
port : 3 dst:00:00:00:00:00:00 src:d0:9f:d9:70:00:e5
port : 4 dst:00:00:00:00:00:00 src:d0:9f:d9:70:00:e6
port : 5 dst:00:00:00:00:00:00 src:d0:9f:d9:70:00:e7
port : 6 dst:00:00:00:00:00:00 src:d0:9f:d9:70:00:e8
port : 7 dst:00:00:00:00:00:00 src:d0:9f:d9:70:00:e9
port : 8 dst:00:00:00:00:00:00 src:64:9d:99:ff:ff:60
port : 9 dst:00:00:00:00:00:00 src:64:9d:99:ff:ff:61
port : 10 dst:00:00:00:00:00:00 src:d0:9f:d9:70:00:e4
port : 11 dst:00:00:00:00:00:00 src:d0:9f:d9:70:00:e5
port : 12 dst:00:00:00:00:00:00 src:d0:9f:d9:70:00:e6
port : 13 dst:00:00:00:00:00:00 src:d0:9f:d9:70:00:e7
port : 14 dst:00:00:00:00:00:00 src:d0:9f:d9:70:00:e8
port : 15 dst:00:00:00:00:00:00 src:d0:9f:d9:70:00:e9
port : 16 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 17 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 18 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 19 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 20 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 21 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 22 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 23 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 24 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 25 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 26 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 27 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 28 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 29 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 30 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 31 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
Total Memory :
MBUF_64 : 196560
MBUF_128 : 98280
MBUF_256 : 73710
MBUF_512 : 73710
MBUF_1024 : 73710
MBUF_2048 : 132080
MBUF_4096 : 32768
MBUF_DP_FLOWS : 4194304
MBUF_GLOBAL_FLOWS : 40960
get_each_core_dp_flows : 131072
Total memory : 1.40 Gbytes
-------------------------------
RX core uses TX queue number 5 on all ports
core, c-port, c-queue, s-port, s-queue, lat-queue
------------------------------------------
1 0 0 1 0 0
2 2 0 3 0 0
3 4 0 5 0 0
4 6 0 7 0 0
5 8 0 9 0 0
6 10 0 11 0 0
7 12 0 13 0 0
8 14 0 15 0 0
9 0 1 1 1 255
10 2 1 3 1 255
11 4 1 5 1 255
12 6 1 7 1 255
13 8 1 9 1 255
14 10 1 11 1 255
15 12 1 13 1 255
16 14 1 15 1 255
17 0 2 1 2 255
18 2 2 3 2 255
19 4 2 5 2 255
20 6 2 7 2 255
21 8 2 9 2 255
22 10 2 11 2 255
23 12 2 13 2 255
24 14 2 15 2 255
25 0 3 1 3 255
26 2 3 3 3 255
27 4 3 5 3 255
28 6 3 7 3 255
29 8 3 9 3 255
30 10 3 11 3 255
31 12 3 13 3 255
32 14 3 15 3 255
-------------------------------
no client generator pool configured, using default pool
no server generator pool configured, using default pool
-- loading cap file cap2/udp_64B_bidirectional.pcap
-- loading cap file cap2/udp_64B_bidirectional.pcap
-- loading cap file cap2/udp_64B_bidirectional.pcap
-- loading cap file cap2/udp_64B_bidirectional.pcap
-- loading cap file cap2/udp_64B_bidirectional.pcap
-- loading cap file cap2/udp_64B_bidirectional.pcap
-- loading cap file cap2/udp_64B_bidirectional.pcap
-- loading cap file cap2/udp_64B_bidirectional.pcap
id name tps cps f-pkts f-bytes duration Mb/sec MB/sec c-flows PPS errors flows
00 cap2/udp_64B_bidirectional.pcap 1000.00 1000.00 2 128 0.00 1.02 0.13 0 2000 0 1
01 cap2/udp_64B_bidirectional.pcap 1000.00 1000.00 2 128 0.00 1.02 0.13 0 2000 0 1
02 cap2/udp_64B_bidirectional.pcap 1000.00 1000.00 2 128 0.00 1.02 0.13 0 2000 0 1
03 cap2/udp_64B_bidirectional.pcap 1000.00 1000.00 2 128 0.00 1.02 0.13 0 2000 0 1
04 cap2/udp_64B_bidirectional.pcap 1000.00 1000.00 2 128 0.00 1.02 0.13 0 2000 0 1
05 cap2/udp_64B_bidirectional.pcap 1000.00 1000.00 2 128 0.00 1.02 0.13 0 2000 0 1
06 cap2/udp_64B_bidirectional.pcap 1000.00 1000.00 2 128 0.00 1.02 0.13 0 2000 0 1
07 cap2/udp_64B_bidirectional.pcap 1000.00 1000.00 2 128 0.00 1.02 0.13 0 2000 0 1
Template #0, extra info:
name : cap2/udp_64B_bidirectional.pcap
cps : 1.000000
ipg : 0.000000
rtt : 0.000000
w : 1
limit : 0
cap_mode : 0
plugin_id : 0
one_server : 0
flows_dirs : []
one_server_was_set : 0
multi_flow_enabled : 0
keep_src_port : 0
ip_header_offset : 0
max_ip_tunnels : 1
Template #1, extra info:
name : cap2/udp_64B_bidirectional.pcap
cps : 1.000000
ipg : 0.000000
rtt : 0.000000
w : 1
limit : 0
cap_mode : 0
plugin_id : 0
one_server : 0
flows_dirs : []
one_server_was_set : 0
multi_flow_enabled : 0
keep_src_port : 0
ip_header_offset : 0
max_ip_tunnels : 1
Template #2, extra info:
name : cap2/udp_64B_bidirectional.pcap
cps : 1.000000
ipg : 0.000000
rtt : 0.000000
w : 1
limit : 0
cap_mode : 0
plugin_id : 0
one_server : 0
flows_dirs : []
one_server_was_set : 0
multi_flow_enabled : 0
keep_src_port : 0
ip_header_offset : 0
max_ip_tunnels : 1
Template #3, extra info:
name : cap2/udp_64B_bidirectional.pcap
cps : 1.000000
ipg : 0.000000
rtt : 0.000000
w : 1
limit : 0
cap_mode : 0
plugin_id : 0
one_server : 0
flows_dirs : []
one_server_was_set : 0
multi_flow_enabled : 0
keep_src_port : 0
ip_header_offset : 0
max_ip_tunnels : 1
Template #4, extra info:
name : cap2/udp_64B_bidirectional.pcap
cps : 1.000000
ipg : 0.000000
rtt : 0.000000
w : 1
limit : 0
cap_mode : 0
plugin_id : 0
one_server : 0
flows_dirs : []
one_server_was_set : 0
multi_flow_enabled : 0
keep_src_port : 0
ip_header_offset : 0
max_ip_tunnels : 1
Template #5, extra info:
name : cap2/udp_64B_bidirectional.pcap
cps : 1.000000
ipg : 0.000000
rtt : 0.000000
w : 1
limit : 0
cap_mode : 0
plugin_id : 0
one_server : 0
flows_dirs : []
one_server_was_set : 0
multi_flow_enabled : 0
keep_src_port : 0
ip_header_offset : 0
max_ip_tunnels : 1
Template #6, extra info:
name : cap2/udp_64B_bidirectional.pcap
cps : 1.000000
ipg : 0.000000
rtt : 0.000000
w : 1
limit : 0
cap_mode : 0
plugin_id : 0
one_server : 0
flows_dirs : []
one_server_was_set : 0
multi_flow_enabled : 0
keep_src_port : 0
ip_header_offset : 0
max_ip_tunnels : 1
Template #7, extra info:
name : cap2/udp_64B_bidirectional.pcap
cps : 1.000000
ipg : 0.000000
rtt : 0.000000
w : 1
limit : 0
cap_mode : 0
plugin_id : 0
one_server : 0
flows_dirs : []
one_server_was_set : 0
multi_flow_enabled : 0
keep_src_port : 0
ip_header_offset : 0
max_ip_tunnels : 1
00 sum 8000.00 64000.00 16 1024 0.00 8.19 1.02 0 16000 0 8
Memory usage
size_64 : 0
size_128 : 0
size_256 : 0
size_512 : 0
size_1024 : 0
size_2048 : 0
size_4096 : 0
size_8192 : 0
size_16384 : 0
Total : 0.00 bytes -nan% util due to buckets
EAL: memzone_reserve_aligned_thread_unsafe(): Number of requested memzone segments exceeds RTE_MAX_MEMZONE
RING: Cannot reserve memory
ERROR there is not enough huge-pages memory in your system
EAL: Error - exiting with code: 1
Cause: Cannot init nodes mbuf pool nodes-27
Here is the run with 2048kB huge pages:
$ cat /proc/meminfo | grep Huge
AnonHugePages: 0 kB
ShmemHugePages: 0 kB
FileHugePages: 0 kB
HugePages_Total: 65536
HugePages_Free: 65529
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
Hugetlb: 134217728 kB
$ ./t-rex-64 -f cap2/imix_64_e810_max.yaml --cfg cfg/e810c.yaml -c 4 -v 7
The ports are bound/configured.
Starting TRex v3.03 please wait ...
Using configuration file cfg/e810c.yaml
port limit : 16
port_bandwidth_gb : 10
port_speed : 0
port_mtu : 0
if_mask : None
is low-end : 0
stack type :
thread_per_dual_if : 1
if : 01:00.0, 01:00.1, 01:00.2, 01:00.3, 01:00.4, 01:00.5, 01:00.6, 01:00.7, 61:00.0, 61:00.1, 61:00.2, 61:00.3, 61:00.4, 61:00.5, 61:00.6, 61:00.7,
enable_zmq_pub : 1
zmq_pub_port : 4500
m_zmq_rpc_port : 4501
src : 00:00:00:00:00:00
dest : 00:00:00:00:00:00
src : 00:00:00:00:00:00
dest : 00:00:00:00:00:00
src : 00:00:00:00:00:00
dest : 00:00:00:00:00:00
src : 00:00:00:00:00:00
dest : 00:00:00:00:00:00
src : 00:00:00:00:00:00
dest : 00:00:00:00:00:00
src : 00:00:00:00:00:00
dest : 00:00:00:00:00:00
src : 00:00:00:00:00:00
dest : 00:00:00:00:00:00
src : 00:00:00:00:00:00
dest : 00:00:00:00:00:00
src : 00:00:00:00:00:00
dest : 00:00:00:00:00:00
src : 00:00:00:00:00:00
dest : 00:00:00:00:00:00
src : 00:00:00:00:00:00
dest : 00:00:00:00:00:00
src : 00:00:00:00:00:00
dest : 00:00:00:00:00:00
src : 00:00:00:00:00:00
dest : 00:00:00:00:00:00
src : 00:00:00:00:00:00
dest : 00:00:00:00:00:00
src : 00:00:00:00:00:00
dest : 00:00:00:00:00:00
src : 00:00:00:00:00:00
dest : 00:00:00:00:00:00
memory per 2x10G ports
MBUF_64 : 16380
MBUF_128 : 8190
MBUF_256 : 8190
MBUF_512 : 8190
MBUF_1024 : 8190
MBUF_2048 : 4095
MBUF_4096 : 128
MBUF_9K : 512
TRAFFIC_MBUF_64 : 65520
TRAFFIC_MBUF_128 : 32760
TRAFFIC_MBUF_256 : 8190
TRAFFIC_MBUF_512 : 8190
TRAFFIC_MBUF_1024 : 8190
TRAFFIC_MBUF_2048 : 32760
TRAFFIC_MBUF_4096 : 128
TRAFFIC_MBUF_9K : 512
MBUF_DP_FLOWS : 524288
MBUF_GLOBAL_FLOWS : 5120
no platform info
CTimerWheelYamlInfo does not exist
flags : 8040f00
write_file : 0
verbose : 7
realtime : 1
flip : 0
cores : 4
single core : 0
flow-flip : 0
no clean close : 0
zmq_publish : 1
vlan mode : 0
client_cfg : 0
mbuf_cache_disable : 0
cfg file : cap2/imix_64_e810_max.yaml
mac file :
out file :
client cfg file :
duration : 3600
factor : 1
mbuf_factor : 1
latency : 0 pkt/sec
zmq_port : 4500
telnet_port : 4501
expected_ports : 16
tw_bucket_usec : 20.000000 usec
tw_buckets : 1024 usec
tw_levels : 3 usec
port : 0 dst:00:00:00:00:00:00 src:00:00:00:00:00:00
port : 1 dst:00:00:00:00:00:00 src:00:00:00:00:00:00
port : 2 dst:00:00:00:00:00:00 src:00:00:00:00:00:00
port : 3 dst:00:00:00:00:00:00 src:00:00:00:00:00:00
port : 4 dst:00:00:00:00:00:00 src:00:00:00:00:00:00
port : 5 dst:00:00:00:00:00:00 src:00:00:00:00:00:00
port : 6 dst:00:00:00:00:00:00 src:00:00:00:00:00:00
port : 7 dst:00:00:00:00:00:00 src:00:00:00:00:00:00
port : 8 dst:00:00:00:00:00:00 src:00:00:00:00:00:00
port : 9 dst:00:00:00:00:00:00 src:00:00:00:00:00:00
port : 10 dst:00:00:00:00:00:00 src:00:00:00:00:00:00
port : 11 dst:00:00:00:00:00:00 src:00:00:00:00:00:00
port : 12 dst:00:00:00:00:00:00 src:00:00:00:00:00:00
port : 13 dst:00:00:00:00:00:00 src:00:00:00:00:00:00
port : 14 dst:00:00:00:00:00:00 src:00:00:00:00:00:00
port : 15 dst:00:00:00:00:00:00 src:00:00:00:00:00:00
port : 16 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 17 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 18 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 19 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 20 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 21 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 22 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 23 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 24 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 25 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 26 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 27 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 28 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 29 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 30 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 31 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
Total Memory :
MBUF_64 : 196560
MBUF_128 : 98280
MBUF_256 : 73710
MBUF_512 : 73710
MBUF_1024 : 73710
MBUF_2048 : 65520
MBUF_4096 : 32768
MBUF_DP_FLOWS : 4194304
MBUF_GLOBAL_FLOWS : 40960
get_each_core_dp_flows : 131072
Total memory : 1.26 Gbytes
there is no configuration file given
DPDK args
xx -l 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33 -n 4 --log-level 8 --main-lcore 0 -a 0000:01:00.0 -a 0000:01:00.1 -a 0000:01:00.2 -a 0000:01:00.3 -a 0000:01:00.4 -a 0000:01:00.5 -a 0000:01:00.6 -a 0000:01:00.7 -a 0000:61:00.0 -a 0000:61:00.1 -a 0000:61:00.2 -a 0000:61:00.3 -a 0000:61:00.4 -a 0000:61:00.5 -a 0000:61:00.6 -a 0000:61:00.7
EAL: Detected CPU lcores: 64
EAL: Detected NUMA nodes: 1
EAL: Detected static linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'VA'
EAL: VFIO support initialized
EAL: Using IOMMU type 1 (Type 1)
EAL: Probe PCI driver: net_ice (8086:1592) device: 0000:01:00.0 (socket 0)
ice_load_pkg_type(): Active package is: 1.3.30.0, ICE OS Default Package (single VLAN mode)
EAL: Probe PCI driver: net_ice (8086:1592) device: 0000:01:00.1 (socket 0)
ice_load_pkg_type(): Active package is: 1.3.30.0, ICE OS Default Package (single VLAN mode)
EAL: Probe PCI driver: net_ice (8086:1592) device: 0000:01:00.2 (socket 0)
ice_load_pkg_type(): Active package is: 1.3.30.0, ICE OS Default Package (single VLAN mode)
EAL: Probe PCI driver: net_ice (8086:1592) device: 0000:01:00.3 (socket 0)
ice_load_pkg_type(): Active package is: 1.3.30.0, ICE OS Default Package (single VLAN mode)
EAL: Probe PCI driver: net_ice (8086:1592) device: 0000:01:00.4 (socket 0)
ice_load_pkg_type(): Active package is: 1.3.30.0, ICE OS Default Package (single VLAN mode)
EAL: Probe PCI driver: net_ice (8086:1592) device: 0000:01:00.5 (socket 0)
ice_load_pkg_type(): Active package is: 1.3.30.0, ICE OS Default Package (single VLAN mode)
EAL: Probe PCI driver: net_ice (8086:1592) device: 0000:01:00.6 (socket 0)
ice_load_pkg_type(): Active package is: 1.3.30.0, ICE OS Default Package (single VLAN mode)
EAL: Probe PCI driver: net_ice (8086:1592) device: 0000:01:00.7 (socket 0)
ice_load_pkg_type(): Active package is: 1.3.30.0, ICE OS Default Package (single VLAN mode)
EAL: Probe PCI driver: net_ice (8086:1592) device: 0000:61:00.0 (socket 0)
ice_load_pkg_type(): Active package is: 1.3.30.0, ICE OS Default Package (single VLAN mode)
EAL: Probe PCI driver: net_ice (8086:1592) device: 0000:61:00.1 (socket 0)
ice_load_pkg_type(): Active package is: 1.3.30.0, ICE OS Default Package (single VLAN mode)
EAL: Probe PCI driver: net_ice (8086:1592) device: 0000:61:00.2 (socket 0)
ice_load_pkg_type(): Active package is: 1.3.30.0, ICE OS Default Package (single VLAN mode)
EAL: Probe PCI driver: net_ice (8086:1592) device: 0000:61:00.3 (socket 0)
ice_load_pkg_type(): Active package is: 1.3.30.0, ICE OS Default Package (single VLAN mode)
EAL: Probe PCI driver: net_ice (8086:1592) device: 0000:61:00.4 (socket 0)
ice_load_pkg_type(): Active package is: 1.3.30.0, ICE OS Default Package (single VLAN mode)
EAL: Probe PCI driver: net_ice (8086:1592) device: 0000:61:00.5 (socket 0)
ice_load_pkg_type(): Active package is: 1.3.30.0, ICE OS Default Package (single VLAN mode)
EAL: Probe PCI driver: net_ice (8086:1592) device: 0000:61:00.6 (socket 0)
ice_load_pkg_type(): Active package is: 1.3.30.0, ICE OS Default Package (single VLAN mode)
EAL: Probe PCI driver: net_ice (8086:1592) device: 0000:61:00.7 (socket 0)
ice_load_pkg_type(): Active package is: 1.3.30.0, ICE OS Default Package (single VLAN mode)
TELEMETRY: No legacy callbacks, legacy socket not created
input : [01:00.0, 01:00.1, 01:00.2, 01:00.3, 01:00.4, 01:00.5, 01:00.6, 01:00.7, 61:00.0, 61:00.1, 61:00.2, 61:00.3, 61:00.4, 61:00.5, 61:00.6, 61:00.7]
dpdk : [0000:01:00.0, 0000:01:00.1, 0000:01:00.2, 0000:01:00.3, 0000:01:00.4, 0000:01:00.5, 0000:01:00.6, 0000:01:00.7, 0000:61:00.0, 0000:61:00.1, 0000:61:00.2, 0000:61:00.3, 0000:61:00.4, 0000:61:00.5, 0000:61:00.6, 0000:61:00.7]
pci_scan : [0000:01:00.0, 0000:01:00.1, 0000:01:00.2, 0000:01:00.3, 0000:01:00.4, 0000:01:00.5, 0000:01:00.6, 0000:01:00.7, 0000:61:00.0, 0000:61:00.1, 0000:61:00.2, 0000:61:00.3, 0000:61:00.4, 0000:61:00.5, 0000:61:00.6, 0000:61:00.7]
map : [ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]
TRex port mapping
-----------------
TRex vport: 0 dpdk_rte_eth: 0
TRex vport: 1 dpdk_rte_eth: 1
TRex vport: 2 dpdk_rte_eth: 2
TRex vport: 3 dpdk_rte_eth: 3
TRex vport: 4 dpdk_rte_eth: 4
TRex vport: 5 dpdk_rte_eth: 5
TRex vport: 6 dpdk_rte_eth: 6
TRex vport: 7 dpdk_rte_eth: 7
TRex vport: 8 dpdk_rte_eth: 8
TRex vport: 9 dpdk_rte_eth: 9
TRex vport: 10 dpdk_rte_eth: 10
TRex vport: 11 dpdk_rte_eth: 11
TRex vport: 12 dpdk_rte_eth: 12
TRex vport: 13 dpdk_rte_eth: 13
TRex vport: 14 dpdk_rte_eth: 14
TRex vport: 15 dpdk_rte_eth: 15
set driver name net_ice
driver capability : TCP_UDP_OFFLOAD TSO SLRO
set dpdk queues mode to DROP_QUE_FILTER
DPDK devices 16 : 16
-----
0 : vdev 0000:01:00.0
1 : vdev 0000:01:00.1
2 : vdev 0000:01:00.2
3 : vdev 0000:01:00.3
4 : vdev 0000:01:00.4
5 : vdev 0000:01:00.5
6 : vdev 0000:01:00.6
7 : vdev 0000:01:00.7
8 : vdev 0000:61:00.0
9 : vdev 0000:61:00.1
10 : vdev 0000:61:00.2
11 : vdev 0000:61:00.3
12 : vdev 0000:61:00.4
13 : vdev 0000:61:00.5
14 : vdev 0000:61:00.6
15 : vdev 0000:61:00.7
-----
Number of ports found: 16
if_index : 0
driver name : net_ice
min_rx_bufsize : 1024
max_rx_pktlen : 9728
max_rx_queues : 64
max_tx_queues : 64
max_mac_addrs : 64
rx_offload_capa : 0x9666f
tx_offload_capa : 0x1181bf
rss reta_size : 512
flow_type_rss : 0x7ffc
tx_desc_max : 4096
tx_desc_min : 64
rx_desc_max : 4096
rx_desc_min : 64
zmq publisher at: tcp://*:4500
rx_data_q_num : 1
rx_drop_q_num : 1
rx_dp_q_num : 0
rx_que_total : 2
--
rx_desc_num_data_q : 4096
rx_desc_num_drop_q : 64
rx_desc_num_dp_q : 4096
total_desc : 4160
--
tx_desc_num : 1024
port 0 desc: Ethernet Controller E810-C for QSFP
rx_qid: 0 (64)
rx_qid: 1 (4096)
ice_set_rx_function(): Using AVX2 Vector Scattered Rx (port 0).
port 1 desc: Ethernet Controller E810-C for QSFP
rx_qid: 0 (64)
rx_qid: 1 (4096)
ice_set_rx_function(): Using AVX2 Vector Scattered Rx (port 1).
port 2 desc: Ethernet Controller E810-C for QSFP
rx_qid: 0 (64)
rx_qid: 1 (4096)
ice_set_rx_function(): Using AVX2 Vector Scattered Rx (port 2).
port 3 desc: Ethernet Controller E810-C for QSFP
rx_qid: 0 (64)
rx_qid: 1 (4096)
ice_set_rx_function(): Using AVX2 Vector Scattered Rx (port 3).
port 4 desc: Ethernet Controller E810-C for QSFP
rx_qid: 0 (64)
rx_qid: 1 (4096)
ice_set_rx_function(): Using AVX2 Vector Scattered Rx (port 4).
port 5 desc: Ethernet Controller E810-C for QSFP
rx_qid: 0 (64)
rx_qid: 1 (4096)
ice_set_rx_function(): Using AVX2 Vector Scattered Rx (port 5).
port 6 desc: Ethernet Controller E810-C for QSFP
rx_qid: 0 (64)
rx_qid: 1 (4096)
ice_set_rx_function(): Using AVX2 Vector Scattered Rx (port 6).
port 7 desc: Ethernet Controller E810-C for QSFP
rx_qid: 0 (64)
rx_qid: 1 (4096)
ice_set_rx_function(): Using AVX2 Vector Scattered Rx (port 7).
port 8 desc: Ethernet Controller E810-C for QSFP
rx_qid: 0 (64)
rx_qid: 1 (4096)
ice_set_rx_function(): Using AVX2 Vector Scattered Rx (port 8).
port 9 desc: Ethernet Controller E810-C for QSFP
rx_qid: 0 (64)
rx_qid: 1 (4096)
ice_set_rx_function(): Using AVX2 Vector Scattered Rx (port 9).
port 10 desc: Ethernet Controller E810-C for QSFP
rx_qid: 0 (64)
rx_qid: 1 (4096)
ice_set_rx_function(): Using AVX2 Vector Scattered Rx (port 10).
port 11 desc: Ethernet Controller E810-C for QSFP
rx_qid: 0 (64)
rx_qid: 1 (4096)
ice_set_rx_function(): Using AVX2 Vector Scattered Rx (port 11).
port 12 desc: Ethernet Controller E810-C for QSFP
rx_qid: 0 (64)
rx_qid: 1 (4096)
ice_set_rx_function(): Using AVX2 Vector Scattered Rx (port 12).
port 13 desc: Ethernet Controller E810-C for QSFP
rx_qid: 0 (64)
rx_qid: 1 (4096)
ice_set_rx_function(): Using AVX2 Vector Scattered Rx (port 13).
port 14 desc: Ethernet Controller E810-C for QSFP
rx_qid: 0 (64)
rx_qid: 1 (4096)
ice_set_rx_function(): Using AVX2 Vector Scattered Rx (port 14).
port 15 desc: Ethernet Controller E810-C for QSFP
rx_qid: 0 (64)
rx_qid: 1 (4096)
ice_set_rx_function(): Using AVX2 Vector Scattered Rx (port 15).
wait 1 sec .
port : 0
------------
link : link : Link Up - speed 10000 Mbps - full-duplex
promiscuous : 0
port : 1
------------
link : link : Link Up - speed 10000 Mbps - full-duplex
promiscuous : 0
port : 2
------------
link : link : Link Up - speed 10000 Mbps - full-duplex
promiscuous : 0
port : 3
------------
link : link : Link Up - speed 10000 Mbps - full-duplex
promiscuous : 0
port : 4
------------
link : link : Link Up - speed 10000 Mbps - full-duplex
promiscuous : 0
port : 5
------------
link : link : Link Up - speed 10000 Mbps - full-duplex
promiscuous : 0
port : 6
------------
link : link : Link Up - speed 10000 Mbps - full-duplex
promiscuous : 0
port : 7
------------
link : link : Link Up - speed 10000 Mbps - full-duplex
promiscuous : 0
port : 8
------------
link : link : Link Up - speed 10000 Mbps - full-duplex
promiscuous : 0
port : 9
------------
link : link : Link Up - speed 10000 Mbps - full-duplex
promiscuous : 0
port : 10
------------
link : link : Link Up - speed 10000 Mbps - full-duplex
promiscuous : 0
port : 11
------------
link : link : Link Up - speed 10000 Mbps - full-duplex
promiscuous : 0
port : 12
------------
link : link : Link Up - speed 10000 Mbps - full-duplex
promiscuous : 0
port : 13
------------
link : link : Link Up - speed 10000 Mbps - full-duplex
promiscuous : 0
port : 14
------------
link : link : Link Up - speed 10000 Mbps - full-duplex
promiscuous : 0
port : 15
------------
link : link : Link Up - speed 10000 Mbps - full-duplex
promiscuous : 0
number of ports : 16
max cores for 2 ports : 4
tx queues per port : 6
no client generator pool configured, using default pool
no server generator pool configured, using default pool
flags : 8040f00
write_file : 0
verbose : 7
realtime : 1
flip : 0
cores : 4
single core : 0
flow-flip : 0
no clean close : 0
zmq_publish : 1
vlan mode : 0
client_cfg : 0
mbuf_cache_disable : 0
cfg file : cap2/imix_64_e810_max.yaml
mac file :
out file :
client cfg file :
duration : 3600
factor : 1
mbuf_factor : 1
latency : 0 pkt/sec
zmq_port : 4500
telnet_port : 4501
expected_ports : 16
tw_bucket_usec : 20.000000 usec
tw_buckets : 1024 usec
tw_levels : 3 usec
port : 0 dst:00:00:00:00:00:00 src:64:9d:99:ff:f2:be
port : 1 dst:00:00:00:00:00:00 src:64:9d:99:ff:f2:bf
port : 2 dst:00:00:00:00:00:00 src:d0:9f:d9:70:00:e4
port : 3 dst:00:00:00:00:00:00 src:d0:9f:d9:70:00:e5
port : 4 dst:00:00:00:00:00:00 src:d0:9f:d9:70:00:e6
port : 5 dst:00:00:00:00:00:00 src:d0:9f:d9:70:00:e7
port : 6 dst:00:00:00:00:00:00 src:d0:9f:d9:70:00:e8
port : 7 dst:00:00:00:00:00:00 src:d0:9f:d9:70:00:e9
port : 8 dst:00:00:00:00:00:00 src:64:9d:99:ff:ff:60
port : 9 dst:00:00:00:00:00:00 src:64:9d:99:ff:ff:61
port : 10 dst:00:00:00:00:00:00 src:d0:9f:d9:70:00:e4
port : 11 dst:00:00:00:00:00:00 src:d0:9f:d9:70:00:e5
port : 12 dst:00:00:00:00:00:00 src:d0:9f:d9:70:00:e6
port : 13 dst:00:00:00:00:00:00 src:d0:9f:d9:70:00:e7
port : 14 dst:00:00:00:00:00:00 src:d0:9f:d9:70:00:e8
port : 15 dst:00:00:00:00:00:00 src:d0:9f:d9:70:00:e9
port : 16 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 17 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 18 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 19 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 20 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 21 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 22 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 23 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 24 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 25 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 26 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 27 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 28 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 29 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 30 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 31 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
Total Memory :
MBUF_64 : 196560
MBUF_128 : 98280
MBUF_256 : 73710
MBUF_512 : 73710
MBUF_1024 : 73710
MBUF_2048 : 132080
MBUF_4096 : 32768
MBUF_DP_FLOWS : 4194304
MBUF_GLOBAL_FLOWS : 40960
get_each_core_dp_flows : 131072
Total memory : 1.40 Gbytes
-------------------------------
RX core uses TX queue number 5 on all ports
core, c-port, c-queue, s-port, s-queue, lat-queue
------------------------------------------
1 0 0 1 0 0
2 2 0 3 0 0
3 4 0 5 0 0
4 6 0 7 0 0
5 8 0 9 0 0
6 10 0 11 0 0
7 12 0 13 0 0
8 14 0 15 0 0
9 0 1 1 1 255
10 2 1 3 1 255
11 4 1 5 1 255
12 6 1 7 1 255
13 8 1 9 1 255
14 10 1 11 1 255
15 12 1 13 1 255
16 14 1 15 1 255
17 0 2 1 2 255
18 2 2 3 2 255
19 4 2 5 2 255
20 6 2 7 2 255
21 8 2 9 2 255
22 10 2 11 2 255
23 12 2 13 2 255
24 14 2 15 2 255
25 0 3 1 3 255
26 2 3 3 3 255
27 4 3 5 3 255
28 6 3 7 3 255
29 8 3 9 3 255
30 10 3 11 3 255
31 12 3 13 3 255
32 14 3 15 3 255
-------------------------------
no client generator pool configured, using default pool
no server generator pool configured, using default pool
-- loading cap file cap2/udp_64B_bidirectional.pcap
-- loading cap file cap2/udp_64B_bidirectional.pcap
-- loading cap file cap2/udp_64B_bidirectional.pcap
-- loading cap file cap2/udp_64B_bidirectional.pcap
-- loading cap file cap2/udp_64B_bidirectional.pcap
-- loading cap file cap2/udp_64B_bidirectional.pcap
-- loading cap file cap2/udp_64B_bidirectional.pcap
-- loading cap file cap2/udp_64B_bidirectional.pcap
id name tps cps f-pkts f-bytes duration Mb/sec MB/sec c-flows PPS errors flows
00 cap2/udp_64B_bidirectional.pcap 1000.00 1000.00 2 128 0.00 1.02 0.13 0 2000 0 1
01 cap2/udp_64B_bidirectional.pcap 1000.00 1000.00 2 128 0.00 1.02 0.13 0 2000 0 1
02 cap2/udp_64B_bidirectional.pcap 1000.00 1000.00 2 128 0.00 1.02 0.13 0 2000 0 1
03 cap2/udp_64B_bidirectional.pcap 1000.00 1000.00 2 128 0.00 1.02 0.13 0 2000 0 1
04 cap2/udp_64B_bidirectional.pcap 1000.00 1000.00 2 128 0.00 1.02 0.13 0 2000 0 1
05 cap2/udp_64B_bidirectional.pcap 1000.00 1000.00 2 128 0.00 1.02 0.13 0 2000 0 1
06 cap2/udp_64B_bidirectional.pcap 1000.00 1000.00 2 128 0.00 1.02 0.13 0 2000 0 1
07 cap2/udp_64B_bidirectional.pcap 1000.00 1000.00 2 128 0.00 1.02 0.13 0 2000 0 1
Template #0, extra info:
name : cap2/udp_64B_bidirectional.pcap
cps : 1.000000
ipg : 0.000000
rtt : 0.000000
w : 1
limit : 0
cap_mode : 0
plugin_id : 0
one_server : 0
flows_dirs : []
one_server_was_set : 0
multi_flow_enabled : 0
keep_src_port : 0
ip_header_offset : 0
max_ip_tunnels : 1
Template #1, extra info:
name : cap2/udp_64B_bidirectional.pcap
cps : 1.000000
ipg : 0.000000
rtt : 0.000000
w : 1
limit : 0
cap_mode : 0
plugin_id : 0
one_server : 0
flows_dirs : []
one_server_was_set : 0
multi_flow_enabled : 0
keep_src_port : 0
ip_header_offset : 0
max_ip_tunnels : 1
Template #2, extra info:
name : cap2/udp_64B_bidirectional.pcap
cps : 1.000000
ipg : 0.000000
rtt : 0.000000
w : 1
limit : 0
cap_mode : 0
plugin_id : 0
one_server : 0
flows_dirs : []
one_server_was_set : 0
multi_flow_enabled : 0
keep_src_port : 0
ip_header_offset : 0
max_ip_tunnels : 1
Template #3, extra info:
name : cap2/udp_64B_bidirectional.pcap
cps : 1.000000
ipg : 0.000000
rtt : 0.000000
w : 1
limit : 0
cap_mode : 0
plugin_id : 0
one_server : 0
flows_dirs : []
one_server_was_set : 0
multi_flow_enabled : 0
keep_src_port : 0
ip_header_offset : 0
max_ip_tunnels : 1
Template #4, extra info:
name : cap2/udp_64B_bidirectional.pcap
cps : 1.000000
ipg : 0.000000
rtt : 0.000000
w : 1
limit : 0
cap_mode : 0
plugin_id : 0
one_server : 0
flows_dirs : []
one_server_was_set : 0
multi_flow_enabled : 0
keep_src_port : 0
ip_header_offset : 0
max_ip_tunnels : 1
Template #5, extra info:
name : cap2/udp_64B_bidirectional.pcap
cps : 1.000000
ipg : 0.000000
rtt : 0.000000
w : 1
limit : 0
cap_mode : 0
plugin_id : 0
one_server : 0
flows_dirs : []
one_server_was_set : 0
multi_flow_enabled : 0
keep_src_port : 0
ip_header_offset : 0
max_ip_tunnels : 1
Template #6, extra info:
name : cap2/udp_64B_bidirectional.pcap
cps : 1.000000
ipg : 0.000000
rtt : 0.000000
w : 1
limit : 0
cap_mode : 0
plugin_id : 0
one_server : 0
flows_dirs : []
one_server_was_set : 0
multi_flow_enabled : 0
keep_src_port : 0
ip_header_offset : 0
max_ip_tunnels : 1
Template #7, extra info:
name : cap2/udp_64B_bidirectional.pcap
cps : 1.000000
ipg : 0.000000
rtt : 0.000000
w : 1
limit : 0
cap_mode : 0
plugin_id : 0
one_server : 0
flows_dirs : []
one_server_was_set : 0
multi_flow_enabled : 0
keep_src_port : 0
ip_header_offset : 0
max_ip_tunnels : 1
00 sum 8000.00 64000.00 16 1024 0.00 8.19 1.02 0 16000 0 8
Memory usage
size_64 : 0
size_128 : 0
size_256 : 0
size_512 : 0
size_1024 : 0
size_2048 : 0
size_4096 : 0
size_8192 : 0
size_16384 : 0
Total : 0.00 bytes -nan% util due to buckets
EAL: memzone_reserve_aligned_thread_unsafe(): Number of requested memzone segments exceeds RTE_MAX_MEMZONE
RING: Cannot reserve memory
ERROR there is not enough huge-pages memory in your system
EAL: Error - exiting with code: 1
Cause: Cannot init nodes mbuf pool nodes-27
hhaim commented
@beyonson the number of cores is limited to 64. you have 16 interfaces and -c 4 would give the maximum cores (64) you need another one to CP and another one to Rx.
I don't think that 64 cores will give better performance than 32, usually the limitation is IO (DDR/PCIe)
beyonson commented
I also thought that at first, then I read more about the -c flag.
The -c flag is the number of cores per pair not per interface.
Per the TRex documentation:
-c
Number of hardware threads to use per interface pair. Use at least 4 for TRex 40Gbs. TRex uses 2 threads for inner needs. Rest of the threads can be used. Maximum number here, can be number of free threads divided by number of interface pairs. For virtual NICs on VM, we always use one thread per interface pair.
When I run the above code with three cores, -c 3, TRex uses 25 cores total. 1 core for the main TRex process, then 3 cores per pair. 16 interfaces, 8 pairs, 3 cores per pair, 3*8 = 24 cores. Therefore, if I run it with 4 cores per pair, it should only use 32 cores.
Perhaps the code that is checking whether the core count requested exceeds max number of cores is incorrect?
beyonson commented
For example, if I change my configuration file to:
- port_limit : 8
version : 2
port_bandwidth_gb : 10
#List of interfaces. Change to suit your setup. Use ./dpdk_setup_ports.py -s to see available options
interfaces : ["01:00.0","01:00.1","01:00.2","01:00.3","01:00.4","01:00.5","01:00.6","01:00.7"]
port_info : # Port IPs. Change to suit your needs. In case of loopback, you can leave as is.
- ip : 1.1.1.1 # port 0
default_gw : 2.2.2.2
- ip : 2.2.2.2 # port 1
default_gw : 1.1.1.1
- ip : 3.3.3.3 # port 2
default_gw : 4.4.4.4
- ip : 4.4.4.4 # port 3
default_gw : 3.3.3.3
- ip : 5.5.5.5 # port 4
default_gw : 6.6.6.6
- ip : 6.6.6.6 # port 5
default_gw : 5.5.5.5
- ip : 7.7.7.7 # port 6
default_gw : 8.8.8.8
- ip : 8.8.8.8 # port 7
default_gw : 7.7.7.7
Then run ./t-rex-64 -f cap2/imix_1518_e810_max.yaml --cfg cfg/e810c-half.yaml -c 8 -v 7
It runs perfectly fine, and htop shows 33 cores being used, which is expected. 8 interfaces, 4 pairs, 8 cores per pair, 8*4=32 (+1 for TRex).
jpatidar01 commented
@beyonson did you found any solution or workaround for this issue ?
beyonson commented
@jpatidar01 I have not, unfortunately. I am still occasionally working on it though and will post a solution if I find it.