datenlord/async-rdma

Support working with existing apps

wangrunji0408 opened this issue · 6 comments

wangrunji0408 commented

The current implementation defines its own protocol to establish connections and send/receive messages. This brings a limitation that both client and server must use this library to communicate. We can not use this library to build a client of an existing rdma server. So I have 2 suggestions that might help:

  1. Support setting up connections through RDMA CM.
  2. Add lower-level APIs to send/receive raw messages.
Nugine commented

There is a new crate rdma for low-level rdma control.
It doesn't support RDMA CM yet because I can not run the examples in librdmacm successfully. (Ubuntu 20.04, RXE, latest rdma-core).
It supports send/recv now and has RC/UD pingpong examples.

async-rdma is a high-level library. It may be not suitable for your use cases.

wangrunji0408 commented

Looks cool!

Would the rdma crate be async-style? Or just a thin wrapper over rdma-sys?
In other words, what's the difference between rdma and async-rdma?

What I want is still a high-level async wrapper like async-rdma, but following the standard protocol.

Nugine commented

rdma aims to be a rust-style wrapper over the C api. (types, methods and resource management)

async-rdma is

A framework for writing RDMA applications with high-level abstraction and asynchronous APIs.

I think its design is not generic enough for working with existing apps now.

rogercloud commented

Good advices,

  1. To support RDMA CM establishing connection requires a RDMA CM binding, and we also need to check the compatibility with IBV APIs. However, it's a good and correct direction, so we'll try to find a way to implement it.

  2. "Add lower-level APIs to send/receive raw messages." is also a reasonable requirement. The current implementation is too high-level, we'll check the possible "memory safety" issues and provide the lower-level APIs if possible.

GTwhy commented

Agree with your advices.
The compatibility with existing RDMA apps has not been considered in the current implementation and I'm going to try.

GTwhy commented

@wangrunji0408 Hi, we add some APIs for RDMA CM and send/recv raw data in #78 , but I forget ref this issue in commit message. Here is an example.

  • 🎉1