Support for Soft-RoCE?

[ethercflow]

I'm testing Soft-RoCE on CentOS 7.6, and want to try test our database with libvma. I tested iperf3 first, but got error:

[root@localhost ~]# LD_PRELOAD=libvma.so iperf3 -c 192.168.152.134
 VMA INFO: ---------------------------------------------------------------------------
 VMA INFO: VMA_VERSION: 8.7.5-0 Development Snapshot built on Aug  9 2019 00:36:56
 VMA INFO: Cmd Line: iperf3 -c 192.168.152.134
 VMA INFO: Current Time: Mon Mar 22 10:44:18 2021
 VMA INFO: Pid: 75933
 VMA INFO: Architecture: x86_64
 VMA INFO: Node: localhost.localdomain
 VMA INFO: ---------------------------------------------------------------------------
 VMA INFO: Log Level                      INFO                       [VMA_TRACELEVEL]
 VMA INFO: ---------------------------------------------------------------------------
 VMA WARNING: **************************************************************
 VMA WARNING: * NO IMMEDIATE ACTION NEEDED!
 VMA WARNING: * Not enough hugepage resources for VMA memory allocation.
 VMA WARNING: * VMA will continue working with regular memory allocation.
 VMA INFO:    * Optional:
 VMA INFO:    *   1. Switch to a different memory allocation type
 VMA INFO:    *      (VMA_MEM_ALLOC_TYPE!= 2)
 VMA INFO:    *   2. Restart process after increasing the number of
 VMA INFO:    *      hugepages resources in the system:
 VMA INFO:    *      "echo 1000000000 > /proc/sys/kernel/shmmax"
 VMA INFO:    *      "echo 800 > /proc/sys/vm/nr_hugepages"
 VMA WARNING: * Please refer to the memory allocation section in the VMA's
 VMA WARNING: * User Manual for more information
 VMA WARNING: **************************************************************
 VMA ERROR: rfs[0x2249740]:280:create_ibv_flow() Create of QP flow ID (tag: 0) failed with flow dst:192.168.152.135:58985, src:192.168.152.134:5201, proto:TCP (errno=38 - Function not implemented)
 VMA ERROR: ring_simple[0x2226de0]:540:attach_flow() attach_flow=0 failed!
Connecting to host 192.168.152.134, port 5201
 VMA ERROR: rfs[0x2249760]:280:create_ibv_flow() Create of QP flow ID (tag: 0) failed with flow dst:192.168.152.135:43487, src:192.168.152.134:5201, proto:TCP (errno=38 - Function not implemented)
 VMA ERROR: ring_simple[0x2226840]:540:attach_flow() attach_flow=0 failed!

So I want to know:

1. whether libvma support SoftRoce or not?
2. For real RoCE, libvma support all socket actions? eg: non-block write/read, is there any limits? Our database uses grpc.

[root@localhost ~]# rxe_cfg status           // server
  Name        Link  Driver  Speed  NMTU  IPv4_addr        RDEV  RMTU
  ens33       yes   e1000          1500  192.168.152.134  rxe0  1024  (3)
  virbr0      no    bridge         1500  192.168.122.1
  virbr0-nic  no    tun            1500

[root@localhost ~]# rxe_cfg status         // client
  Name        Link  Driver  Speed  NMTU  IPv4_addr        RDEV  RMTU
  ens33       yes   e1000          1500  192.168.152.135  rxe0  1024  (3)
  virbr0      no    bridge         1500  192.168.122.1
  virbr0-nic  no    tun            1500

Version:
libvma-8.7.5-1.el7.x86_64
3.10.0-1160.21.1.el7.x86_64/CentOS 7.9

firewall is disabled.

Thanks a lot!

[ethercflow]

Hi @NirNitzani , @igor-ivanov , is it convenient to take a look when you are free? Thank you!

[igor-ivanov]

Hello @ethercflow,

1. SoftRoce is out of VMA scope
2. VMA supports UDP/TCP, blocking and non-blocking sockets. You can find details at User Manual https://docs.mellanox.com/spaces/viewspace.action?key=VMAv922

[ethercflow]

Got it, thx!