Strange performance problems when going below 6 cores per dual port
Civil opened this issue · 0 comments
Civil commented
I'm currently trying to benchmark NICs on a system that have relatively low core count (16 cores, 32 threads) and I encounter a strange scalability problem when I'm trying to allocate less than 6 cores per dual port.
With 6 cores per dual port on 5 NICs I can easily get about 100 Mpps per NIC TX (and 65-85 Mpps RX, depending on a NIC, which is in line with what )
This is performance on ConnectX-6 with 6 cores per dual-port:
vtune hotspots:
vtune: Executing actions 75 % Generating a report Elapsed Time: 68.307s
CPU Time: 341.346s
Effective Time: 341.346s
Spin Time: 0s
Overhead Time: 0s
Total Thread Count: 17
Paused Time: 0s
Top Hotspots
Function Module CPU Time % of CPU Time(%)
-------------------------------------------------------------------------------------------------------- ------------- -------- ----------------
rte_rdtsc _t-rex-64 132.300s 38.8%
std::priority_queue<CGenNode*, std::vector<CGenNode*, std::allocator<CGenNode*>>, CGenNodeCompare>::push _t-rex-64 38.642s 11.3%
mlx5_tx_burst_empw_inline libmlx5-64.so 19.090s 5.6%
mlx5_tx_cseg_init libmlx5-64.so 15.152s 4.4%
CNodeGenerator::handle_stl_node _t-rex-64 11.840s 3.5%
[Others] N/A 124.322s 36.4%
Effective Physical Core Utilization: 31.6% (5.049 out of 16)
| The metric value is low, which may signal a poor physical CPU cores
| utilization caused by:
| - load imbalance
| - threading runtime overhead
| - contended synchronization
| - thread/process underutilization
| - incorrect affinity that utilizes logical cores instead of physical
| cores
| Explore sub-metrics to estimate the efficiency of MPI and OpenMP parallelism
| or run the Locks and Waits analysis to identify parallel bottlenecks for
| other parallel runtimes.
|
Effective Logical Core Utilization: 15.8% (5.061 out of 32)
| The metric value is low, which may signal a poor logical CPU cores
| utilization. Consider improving physical core utilization as the first
| step and then look at opportunities to utilize logical cores, which in
| some cases can improve processor throughput and overall performance of
| multi-threaded applications.
|
Collection and Platform Info
Application Command Line: ./_t-rex-64 "-i" "-c" "6" "--cfg" "/etc/trex_single.yaml" "--mlx5-so"
Operating System: 6.5.0-0.deb12.4-amd64 12.5
Computer Name: spr-testbench
Result Size: 13.2 MB
Collection start time: 22:11:36 14/03/2024 UTC
Collection stop time: 22:12:46 14/03/2024 UTC
Collector Type: Event-based counting driver,User-mode sampling and tracing
CPU
Name: Intel(R) Xeon(R) Processor code named Sapphirerapids
Frequency: 3.096 GHz
Logical CPU Count: 32
LLC size: 47.2 MB
Cache Allocation Technology
Level 2 capability: available
Level 3 capability: available
Some information from vtune performance-snapshot:
vtune: Executing actions 75 % Generating a report Elapsed Time: 45.157s
IPC: 2.652
SP GFLOPS: 0.000
DP GFLOPS: 0.562
Average CPU Frequency: 4.918 GHz
Logical Core Utilization: 16.4% (5.255 out of 32)
Physical Core Utilization: 32.8% (5.241 out of 16)
Microarchitecture Usage: 43.4% of Pipeline Slots
Retiring: 43.4% of Pipeline Slots
Light Operations: 38.7% of Pipeline Slots
Heavy Operations: 4.7% of Pipeline Slots
Front-End Bound: 2.2% of Pipeline Slots
Front-End Latency: 0.6% of Pipeline Slots
Front-End Bandwidth: 1.6% of Pipeline Slots
Bad Speculation: 1.2% of Pipeline Slots
Branch Mispredict: 0.8% of Pipeline Slots
Machine Clears: 0.4% of Pipeline Slots
Back-End Bound: 53.2% of Pipeline Slots
Memory Bound: 14.2% of Pipeline Slots
L1 Bound: 2.6% of Clockticks
L2 Bound: 0.0% of Clockticks
L3 Bound: 1.0% of Clockticks
L3 Latency: 0.2% of Clockticks
DRAM Bound: 0.0% of Clockticks
Memory Bandwidth: 0.1% of Clockticks
Memory Latency: 9.2% of Clockticks
Local DRAM: 0.0% of Clockticks
Remote DRAM: 0.0% of Clockticks
Remote Cache: 0.0% of Clockticks
Core Bound: 39.0% of Pipeline Slots
Memory Bound: 14.2% of Pipeline Slots
Cache Bound: 3.7% of Clockticks
DRAM Bound: 0.0% of Clockticks
DRAM Bandwidth Bound: 0.0% of Elapsed Time
NUMA: % of Remote Accesses: 0.0%
Vectorization: 0.0% of Packed FP Operations
Instruction Mix
HP FLOPs: 0.0% of uOps
Packed: 0.0%
128-bit: 0.0%
256-bit: 0.0%
512-bit: 0.0%
Scalar: 0.0%
SP FLOPs: 0.0% of uOps
Packed: 11.4% from SP FP
128-bit: 11.0% from SP FP
256-bit: 0.4% from SP FP
512-bit: 0.0% from SP FP
Scalar: 88.6% from SP FP
DP FLOPs: 0.8% of uOps
Packed: 0.0% from DP FP
128-bit: 0.0% from DP FP
256-bit: 0.0% from DP FP
512-bit: 0.0% from DP FP
Scalar: 100.0% from DP FP
AMX BF16 FLOPs: 0.0% of uOps
x87 FLOPs: 0.0% of uOps
Non-FP: 99.2% of uOps
FP Arith/Mem Rd Instr. Ratio: 0.031
FP Arith/Mem Wr Instr. Ratio: 0.069
PCIe Bandwidth: 13.345 GB/s
PCI Device Class PCIe Bandwidth, GB/s
------------------ --------------------
Network controller 13.345
Bridge 0.000
[Unknown] 0.000
And here is for 5 cores per dualport:
Here it is clearly visible drop in TX from 100 Mpps per port to 25 Mpps (1/4 of original for only one core difference) and performance is WAY less stable (it can briefly go up to 40 Mpps but drops)
From vtune hotspots:
vtune: Executing actions 75 % Generating a report Elapsed Time: 82.541s
CPU Time: 381.038s
Effective Time: 381.038s
Spin Time: 0s
Overhead Time: 0s
Total Thread Count: 16
Paused Time: 0s
Top Hotspots
Function Module CPU Time % of CPU Time(%)
-------------------------------------------------------------------------------------------------------- ------------- -------- ----------------
rte_rdtsc _t-rex-64 295.096s 77.4%
std::priority_queue<CGenNode*, std::vector<CGenNode*, std::allocator<CGenNode*>>, CGenNodeCompare>::push _t-rex-64 16.046s 4.2%
rte_delay_us_block _t-rex-64 6.892s 1.8%
rte_pause _t-rex-64 5.934s 1.6%
check_cqe libmlx5-64.so 4.816s 1.3%
[Others] N/A 52.254s 13.7%
Effective Physical Core Utilization: 29.1% (4.657 out of 16)
| The metric value is low, which may signal a poor physical CPU cores
| utilization caused by:
| - load imbalance
| - threading runtime overhead
| - contended synchronization
| - thread/process underutilization
| - incorrect affinity that utilizes logical cores instead of physical
| cores
| Explore sub-metrics to estimate the efficiency of MPI and OpenMP parallelism
| or run the Locks and Waits analysis to identify parallel bottlenecks for
| other parallel runtimes.
|
Effective Logical Core Utilization: 14.6% (4.668 out of 32)
| The metric value is low, which may signal a poor logical CPU cores
| utilization. Consider improving physical core utilization as the first
| step and then look at opportunities to utilize logical cores, which in
| some cases can improve processor throughput and overall performance of
| multi-threaded applications.
|
Collection and Platform Info
Application Command Line: ./_t-rex-64 "-i" "-c" "5" "--cfg" "/etc/trex_single.yaml" "--mlx5-so"
Operating System: 6.5.0-0.deb12.4-amd64 12.5
Computer Name: spr-testbench
Result Size: 13.7 MB
Collection start time: 22:16:15 14/03/2024 UTC
Collection stop time: 22:17:39 14/03/2024 UTC
Collector Type: Event-based counting driver,User-mode sampling and tracing
CPU
Name: Intel(R) Xeon(R) Processor code named Sapphirerapids
Frequency: 3.096 GHz
Logical CPU Count: 32
LLC size: 47.2 MB
Cache Allocation Technology
Level 2 capability: available
Level 3 capability: available
Composition is completely different and most of time is spent on getting time.
and vtune performance snapshot:
vtune: Executing actions 75 % Generating a report Elapsed Time: 59.648s
IPC: 0.894
SP GFLOPS: 0.000
DP GFLOPS: 0.163
Average CPU Frequency: 4.921 GHz
Logical Core Utilization: 13.4% (4.277 out of 32)
Physical Core Utilization: 26.6% (4.263 out of 16)
Microarchitecture Usage: 19.9% of Pipeline Slots
Retiring: 19.9% of Pipeline Slots
Light Operations: 13.0% of Pipeline Slots
Heavy Operations: 6.8% of Pipeline Slots
Front-End Bound: 2.2% of Pipeline Slots
Front-End Latency: 0.6% of Pipeline Slots
Front-End Bandwidth: 1.6% of Pipeline Slots
Bad Speculation: 0.5% of Pipeline Slots
Branch Mispredict: 0.4% of Pipeline Slots
Machine Clears: 0.0% of Pipeline Slots
Back-End Bound: 77.4% of Pipeline Slots
Memory Bound: 5.9% of Pipeline Slots
L1 Bound: 1.2% of Clockticks
L2 Bound: 0.0% of Clockticks
L3 Bound: 1.1% of Clockticks
L3 Latency: 1.6% of Clockticks
DRAM Bound: 0.0% of Clockticks
Memory Bandwidth: 0.1% of Clockticks
Memory Latency: 3.5% of Clockticks
Local DRAM: 0.0% of Clockticks
Remote DRAM: 0.0% of Clockticks
Remote Cache: 0.0% of Clockticks
Core Bound: 71.5% of Pipeline Slots
Memory Bound: 5.9% of Pipeline Slots
Cache Bound: 2.3% of Clockticks
DRAM Bound: 0.0% of Clockticks
DRAM Bandwidth Bound: 0.0% of Elapsed Time
NUMA: % of Remote Accesses: 0.0%
Vectorization: 0.0% of Packed FP Operations
Instruction Mix
HP FLOPs: 0.0% of uOps
Packed: 0.0%
128-bit: 0.0%
256-bit: 0.0%
512-bit: 0.0%
Scalar: 0.0%
SP FLOPs: 0.0% of uOps
Packed: 24.7% from SP FP
128-bit: 24.7% from SP FP
256-bit: 0.0% from SP FP
512-bit: 0.0% from SP FP
Scalar: 75.3% from SP FP
DP FLOPs: 0.7% of uOps
Packed: 0.0% from DP FP
128-bit: 0.0% from DP FP
256-bit: 0.0% from DP FP
512-bit: 0.0% from DP FP
Scalar: 100.0% from DP FP
AMX BF16 FLOPs: 0.0% of uOps
x87 FLOPs: 0.0% of uOps
Non-FP: 99.3% of uOps
FP Arith/Mem Rd Instr. Ratio: 0.031
FP Arith/Mem Wr Instr. Ratio: 0.072
PCIe Bandwidth: 4.280 GB/s
PCI Device Class PCIe Bandwidth, GB/s
------------------ --------------------
Network controller 4.280
Bridge 0.000
[Unknown] 0.000
That looks like COMPLETELY different workload which should be way less stressful for core (IPC <1 compared to 2.6 before) and that looks like some problem with scheduling logic within trex or some problem within DPDK.