ib_write_bw works fine without -R. But only very small bandwidth when using -R option
chgdragon2023 opened this issue · 1 comments
chgdragon2023 commented
When I use ib_write_bw without -R option, everything looks fine to me. However, if -R is used. only very small bandwidth can be achieved. In addition, when -R option is used, the CPU usage on the server side is very high
root@loving-insect:/home/ocp_user# ib_write_bw -d mlx5_0 -i 1 -D 1 -m 4096 -q 1 -s 1000000 -t 10 --report_gbits
************************************
* Waiting for client to connect... *
************************************
---------------------------------------------------------------------------------------
RDMA_Write BW Test
Dual-port : OFF Device : mlx5_0
Number of qps : 1 Transport type : IB
Connection type : RC Using SRQ : OFF
PCIe relax order: ON
ibv_wr* API : ON
CQ Moderation : 1
Mtu : 4096[B]
Link type : Ethernet
GID index : 3
Max inline data : 0[B]
rdma_cm QPs : OFF
Data ex. method : Ethernet
---------------------------------------------------------------------------------------
local address: LID 0000 QPN 0x2da6 PSN 0x38f9aa RKey 0x201200 VAddr 0x007f67c0644240
GID: 00:00:00:00:00:00:00:00:00:00:255:255:11:00:00:72
remote address: LID 0000 QPN 0x3606 PSN 0xf0086a RKey 0x200c00 VAddr 0x007f0a53d57240
GID: 00:00:00:00:00:00:00:00:00:00:255:255:11:00:00:88
---------------------------------------------------------------------------------------
#bytes #iterations BW peak[Gb/sec] BW average[Gb/sec] MsgRate[Mpps]
1000000 48233 0.00 385.86 0.048233
---------------------------------------------------------------------------------------
root@loving-insect:/home/ocp_user# ib_write_bw -d mlx5_0 -i 1 -D 1 -m 4096 -q 1 -s 1000000 -t 10 --report_gbits -R
************************************
* Waiting for client to connect... *
************************************
---------------------------------------------------------------------------------------
RDMA_Write BW Test
Dual-port : OFF Device : mlx5_0
Number of qps : 1 Transport type : IB
Connection type : RC Using SRQ : OFF
PCIe relax order: ON
ibv_wr* API : ON
CQ Moderation : 1
Mtu : 4096[B]
Link type : Ethernet
GID index : 3
Max inline data : 0[B]
rdma_cm QPs : ON
Data ex. method : rdma_cm
---------------------------------------------------------------------------------------
Waiting for client rdma_cm QP to connect
Please run the same command with the IB/RoCE interface IP
---------------------------------------------------------------------------------------
local address: LID 0000 QPN 0x2da8 PSN 0x4fc3c9
GID: 00:00:00:00:00:00:00:00:00:00:255:255:11:00:00:72
remote address: LID 0000 QPN 0x2c24 PSN 0xac206a
GID: 00:00:00:00:00:00:00:00:00:00:255:255:11:00:00:95
---------------------------------------------------------------------------------------
#bytes #iterations BW peak[Gb/sec] BW average[Gb/sec] MsgRate[Mpps]
1000000 6 0.000000 0.048000 0.000006
---------------------------------------------------------------------------------------
root@loving-insect:/home/ocp_user#
Lyflihai commented
you may use ZTRCC as your congestion control protocol