Mellanox/libvma

Rust TcpStream clone cannot send/recv through libvma

boranby opened this issue · 5 comments

boranby commented

Subject

I have a system setup where listening the UDP stream from one port and send/recv TCP from the other port of my NIC. All of the parts are working except the cloned TcpStream part. Using: https://doc.rust-lang.org/std/net/struct.TcpStream.html#method.try_clone

Issue type

  • Bug report
  • Feature request

Configuration:

  • Product version libvma-9.8.40
  • OS - Redhat 9.2
  • OFED MLNX_OFED_LINUX-23.07-0.5.1.2-rhel9.2-x86_64
  • Hardware - Mellanox MT2894 Family [ConnectX-6 Lx]

Actual behavior:

I am using a separate thread to send and receive heartbeats so I pass the cloned TcpStream to that thread. The main TcpStream can send and receive packets but The cloned TcpStream cannot send and receive any packets, and TcpStream::write_all returns with Broken pipe error. When I don't use libvma and offload with OS, everything works as expected.

Expected behavior:

libvma is expected to work same as the OS to offload the traffic from the cloned TcpStream.

Steps to reproduce:

Server gist: https://gist.github.com/boranby/02464d37e188ba1fe6032afc79ba5d50
Client gist: https://gist.github.com/boranby/787343c64d6c5292369fd722977641ea

Need to update the IP addresses and route the Multicast address via port0 and route the TCP connection via port1.

You should see that login packet and login accepted packets reaches, but heartbeats don't reach.

igor-ivanov commented

Hello @boranby, I think in your application scenario you meet libvma limitations such as VMA does not work as server/client from single process and it has restriction in loopback flow usage.

boranby commented

Hi @igor-ivanov , thanks for your response.

I am running the client and server in two different machines.
Does libvma consider cloning a TcpStream to read/write from two threads as server/client from single process?

igor-ivanov commented

do you mean that application uses thread0 for read and thread1 for write from the same process? Note: it should be ok from libvma.
I noticed 127.0.0.1 ip in application sources you provided.

boranby commented

It is more like

  • thread0 to read/write TCP
  • thread1 to receive UDP multicast and write to TCP
boranby commented

I suspected that TCPStream clone has the problem. When I use only one thread to read/write TCP, it works.