ff_recvmsg not working with unconnected UDP sockets

[SebastianVoit]

Greetings,
I wanted to use F-Stack for transmitting QUIC packets (UDP packets from the view of F-Stack, it being QUIC packets should not bear any weight for this issue).
To do so I initialized F-Stack, replaced the socket calls and adjusted the application loop to work with ff_run().
However, I have come across a strange issue: while sending using ff_sendmsg() works perfectly fine, reception via ff_recvmsg() returns either all zeroes or random data in the iovec.
The application has been tested with Linux sockets where the same setup works without issues.

I use the default F-Stack configuration file.

This issue occurs on both the server and the client, but as the socket setup for the client is much simpler, I present a shortened version of the minimal setup here:

1. sockfd = ff_socket(AF_INET, SOCK_DGRAM, 0);

2. ff_bind(sockfd, sa_local, sizeof(struct sockaddr_in));

3. int on = 1; ff_ioctl(sockfd, FIONBIO, &on);

Afterwards the application calls ff_recvmsg() to receive packets and ff_sendmsg() to send packets.

Additional notes:
The F-Stack client is tested against the server running the Linux version (as packet reception would not work otherwise). The client can send out the packets and the server responds. However, the client only receives garbage data from the call to ff_recvmsg().
Inspecting the pcap files generated by DPDK indicate that the packet is received correctly.
ff_recvfrom() properly returns the received data when used in the exact same context (the application requires ancillary messages so this is not an option).
As ff_recvfrom() works correctly with the socket setup and the identical setup works with Linux sockets (in the test setup the server runs the Linux version), I can only conclude that the cause of the issue is a bug inside the ff_recvmsg() function.
However I can never exclude it being a me problem, if so any help would be appreciated.

Thanks,
Sebastian

[freak82]

Hi there,

I hit this problem before in our application which uses F-stack.
The problem is that the ff_recvmsg receives the Linux type of struct msghdr but passes it to the FreeBSD functions as if it's the FreeBSD kind of struct msghdr.
See this issue which I've opened before but I think the fix there is not enough.
The struct msghdr of FreeBSD has msg_iovlen and msg_controllen as 32 bit values while the Linux struct msghdr has these fields as size_t which is 64 bit. And I think this is what causes you application to get garbage when call this function.
I think the fix for the above issue is not enough because the Linux msghdr is still passed to the internal FreeBSD kern_recvit function which expects the FreeBSD msghdr and treats it like such. See the function ff_recvmsg in the ff_syscall_wrapper.c.

In our code base I've fixed the issue by providing the following structure

struct ff_msghdr {                                                              
    void*         msg_name;                                                     
    uint32_t      msg_namelen;                                                  
    struct iovec* msg_iov;                                                      
    int32_t       msg_iovlen;                                                   
    void*         msg_control;                                                  
    uint32_t      msg_controllen;                                               
    int32_t       msg_flags;                                                    
};

and I've changed the ff_recvmsg to accept and use this structure. The above struct corresponds to the FreeBSD struct msghdr.

HTH,
Pavel.

[freak82]

One additional note.
You mentioned that you need to use the ancillary data from struct msghdr.
I think there will be more incompatibilities there between the Linux and the FreeBSD. I mean that the struct cmsghdr and the corresponding macros (CMSG_FIRSTHDR, ...) may be different in FreeBSD and Linux and this can cause further issues. You may need to expose and use struct cmdghdr which correspond to the FreeBSD one and not use the Linux one. However, I haven't dig into this because I've no use case with the ancillary data.

[sarosharif]

Hi,
I am also trying to run QUIC with fstack, and having issues with cmsghdr
Looks like there is a difference between cmsghdr length between linux and freebsd.
In linux:

size_t cmsg_len; /* Length of data in cmsg_data plus length
         of cmsghdr structure.
         !! The type should be socklen_t but the
         definition of the kernel is incompatible
         with this.  */

int cmsg_level;  /* Originating protocol.  */

int cmsg_type;   /* Protocol specific type.  */

In freebsd:

socklen_t	cmsg_len;		/* data byte count, including hdr */

int		cmsg_level;		/* originating protocol */

int		cmsg_type;		/* protocol-specific type */

which causes a 4 byte difference in in total length of the header and hence it's parsing within the code can not work correctly.

Similarly there is a difference in flags provided in cmsghdr provided in linux and freebsd. Such as:
in linux IP_TOS 1 and in freebsd IP_TOS 3 which causes errors like EPROTONOSUPPORT to occur
Do you have any fix for this?

[freak82]

Hi,

I don't have fix for these because I haven't used this functionality myself.
As far as I just checked the code in (ff_syscall_wrapper.c)[https://github.com/F-Stack/f-stack/blob/dev/lib/ff_syscall_wrapper.c] the IP_TOS is handled correctly for the cases of ff_setsockopt/ff_getsockopt but it's not handled correctly for the case of the cmsghdr struct - see freebsd2linux_cmsghdr implementation in the above file.

[jfb8856606]

It will fixed later, and include the pr #775 that has be reverted.

[jfb8856606]

You can test use the lastest code of dev branch to test it.