Behavior of SilkMarshal.StringToPtr and related methods on Linux

[Exanite]

Summary

DirectXShaderCompiler uses wchar_t for its LPWSTRs, which has a different size on Windows (2 bytes) when compared to Linux (4 bytes).
This breaks code that uses SilkMarshal.StringToPtr and similar methods when on Linux.

https://github.com/microsoft/DirectXShaderCompiler/blob/a03a77f7efa49eabe341b331b9066e60ae9aadce/include/dxc/WinAdapter.h#L333-L337

Steps to reproduce

I don't have reproduction code, but this is the code I use in my codebase:

var arguments = new List<string>
{
    // Entrypoint
    "-E", "main",

    // Target profile / shader type + version
    "-T", "ps_6_6",
};

var ppArguments = SilkMarshal.StringArrayToPtr(arguments, NativeStringEncoding.LPWStr);

// ppArguments is then passed into `IDxcCompiler3.Compile()`

When using this code to print out the byte data of ppArguments, the output looks like this:

var bytes = (byte**)ppArguments;
for (var i = 0; i < arguments.Count; i++)
{
    Console.WriteLine($"Argument {i}");
    var j = 0;
    while (true)
    {
        var sum = 0;
        sum += bytes[i][j + 0];
        sum += bytes[i][j + 1];

        Console.WriteLine(bytes[i][j + 0].ToString("x2"));
        Console.WriteLine(bytes[i][j + 1].ToString("x2"));

        j += 2;

        if (sum == 0)
        {
            break;
        }
    }
}

This argument in particular corresponds to "-E". Note that each character is 2 bytes, including the null terminator.
Also note that this is correct behavior on Windows: https://learn.microsoft.com/en-us/openspecs/windows_protocols/ms-dtyp/50e9ef83-d6fd-4e22-a34a-2c6b4e3c24f3

The LPWSTR type is a 32-bit pointer to a string of 16-bit Unicode characters, which MAY be null-terminated.

This StackOverflow post talks about the size different between different platforms of wchar_t: https://stackoverflow.com/questions/13509733/what-is-the-use-of-wchar-t-in-general-programming
Similar posts can also be found.

---

What confirms this difference is when I manually widen the characters by iterating through the bytes returned by StringArrayToPtr and storing the result in a new block of memory.
This converts the 2-byte width characters into 4-byte width characters.

This allows my shader compilation code to work up until I convert a LPWSTR pointer into a C# string by using SilkMarshal.PtrToString.
This is almost definitely the same issue.

Comments

I'm using the libdxcompiler.so binary from the DirectXShaderCompiler repo: https://github.com/microsoft/DirectXShaderCompiler/releases

The code I'm using works perfectly fine on Windows, but breaks on Linux. I've tested this on WSL and NixOS.

The length of wchar_t seems to be implementation dependent and doesn't seem to be simply 2-bytes on Windows and 4-bytes on Linux. I'm not familiar with how to check for this difference on different platforms.

I'm also not sure how endianness plays into this. Endianness might have to be accounted for as well.

Because of these two issues, while I can work around the issue on the current platforms I've tested on, my work around might break again on other platforms. Any suggestions here are welcome!

Changing the behavior of StringToPtr and similar methods to return 4-byte characters for LPWSTR is a breaking change.
The current behavior and "fixed" behavior are both technically correct. I'd argue the 4-byte behavior is more correct, but I'm not sure how LPWSTR is used in other places in Silk.NET.

[Exanite]

Currently working on this issue.

Shouldn't the LPWStr case return 2 (4 on non-Windows) times the length?

Silk.NET/src/Core/Silk.NET.Core/Native/SilkMarshal.cs

Lines 224 to 240 in 8da3988

	/// <summary>
	/// Allocates a string pointer
	/// </summary>
	/// <param name="length">Length of the memory to be allocated</param>
	/// <param name="encoding">The encoding of the string stored in this memory region</param>
	/// <returns>The pointer to the allocated memory</returns>
	/// <seealso cref="FreeString"/>
	public static nint AllocateString(int length, NativeStringEncoding encoding = NativeStringEncoding.Ansi)
	=> encoding switch
	{
	NativeStringEncoding.BStr => AllocBStr(length),
	NativeStringEncoding.LPStr => Allocate(length),
	NativeStringEncoding.LPTStr => Allocate(length),
	NativeStringEncoding.LPUTF8Str => Allocate(length),
	NativeStringEncoding.LPWStr => Allocate(length),
	_ => ThrowInvalidEncoding<nint>()
	};

[Perksey]

AllocateString is mostly to unify Allocate and AllocBStr. The length parameter here represents the byte length, not the character length i.e. it is assumed the caller has already done that multiplication.

[Exanite]

I see! Thank you for the clarification.