issues with ABIs and struct return types

[vvuk]

With the MSVC x64 ABI, structs are returned in RAX if and only if they are <= 8 bytes in size, and are effectively a POD type. See https://msdn.microsoft.com/en-us/library/7572ztz4.aspx . Unfortunately, I can't figure out how to tell Rust this. In C++, sk_sp<T> has a user-defined constructor, destructor, etc. In Rust, I couldn't figure out any way to do this. It's defined as..

    #[repr(C)]
    #[derive(Debug, Copy, Clone)]
    pub struct sk_sp<T> {
        pub ptr: *mut T,
    }

This causes segfaults because the C++ calling convention isn't upheld. A temporary workaround is to just add another dummy field, making this type bigger than 8 bytes. This happens to work in this case since this type is only ever used as a smart pointer return type; params are passed as basic T*.

Is there any way to tell rustc to treat this type as non-POD when giving it to LLVM? If not, can such a mechanism be added?

[retep998]

Actually, this difference extends to 32bit as well. C++ methods have that slight difference in how they handle return types compared to C/C++ functions. A case where I encountered this issue was actually in binding COM interfaces. 99% of the time they'd just be returning HRESULT which is the same for both methods and functions. However every once in a while a COM method would return an aggregate type that is affected by this slight difference and Rust would have the wrong calling convention. Hell, even C compiled by cl.exe using the official C bindings to the COM interface gets it wrong, so this issue is so subtle that not even Microsoft has done anything about it.

Related issue rust-lang/rfcs#1342

[steveklabnik]

Someone ran into this today https://users.rust-lang.org/t/returning-values-from-rust-to-c/11256

EDIT: actually reading more, maybe not...

[comex]

@steveklabnik (Turns out that was indeed this issue: the user added a constructor on the C++ side, making it non-POD, and didn't mention it originally.)

[retep998]

In order to fully solve this we would need two things:

1. Calling conventions for C++ methods rather than functions. The difference in how they handle struct returns is absolutely crucial for code to get right.
2. A new repr for structs to indicate that they are not POD.

[emilio]

This is also an issue for Linux btw, see rust-lang/rust-bindgen#778.

[Ralith]

A new repr for structs to indicate that they are not POD.

The notion of "POD" here is idiosyncratic to the MSVC ABI (e.g. it does not match the C++ notion of "standard-layout"), so care should be taking with naming such a repr to not cause further confusion.