Support for reflection-invoking methods that take pointers by reference

[MichalStrehovsky]

dotnet/coreclr#17732 added support for reflection-invoking methods that return unmanaged pointers by reference (so e.g. ref void* Do()), but methods that get them by-reference as their parameters are still pretty broken (so e.g. void Do(ref void* x)).

For example:

using System;
using System.Reflection;

unsafe class Program
{
    public static void InvokeMe(out int* pointer)
    {
        pointer = (int*)0x12345;
    }

    static void Main(string[] args)
    {
        typeof(Program).GetMethod(nameof(InvokeMe)).Invoke(null, new object[] { null });
    }
}

Will crash the runtime with an ExecutionEngineException.

(Note this behavior is on parity with the desktop CLR, so it has quite likely been this broken for 17 years.)

[MichalStrehovsky]

Cc @atsushikan @sergiy-k

[danmoseley]

@MichalStrehovsky marked future. feel free to change.

[GrabYourPitchforks]

Expected behavior: Once this method returns, the object[] that's passed in contains a single element which is equivalent to having written System.Reflection.Pointer.Box((int*)12345, typeof(int)).

[steveharter]

This does repro on 5.0 - ExecutionEngineException is thrown. Trying to ignore the exception with a try\catch results in Fatal error. Internal CLR error. (0x80131506)

[GrabYourPitchforks]

@MichalStrehovsky Is this actually blocking a critical scenario? A workaround right now would be to define a delegate with an appropriate signature and to wrap that delegate around the target method.

What we want to avoid is spending a bunch of time addressing one-off issues in reflection that very few people will ever realistically run in to. We'd rather prioritize more impactful work items.

[MichalStrehovsky]

@MichalStrehovsky Is this actually blocking a critical scenario?

Nope. If I remember correctly, I found this when we were adding System.Reflection.Pointer support into .NET Native (for NetStandard 2.0, since Pointer was new to NS2.0). I was writing tests and found out this doesn't work so I logged a bug. FWIW I think I made it work as expected in .NET Native.

[steveharter]

See also #7430 which is similar but using a binder.

[steveharter]

In v9, this now crashes with NullReferenceException in System.Reflection.MethodBaseInvoker.InvokeDirectByRefWithFewArgs.