IDCompositionRectangleClip Not Working

Question

IDCompositionRectangleClip Not Working

Closed this issue 3 years ago · 2 comments

I have been porting a C++ direct composition Code to CS, In my C++ code I use IDCompositionRectangleClip to clip the visual, it is working fine with C++, when I am trying to implement it in CS using DirectN, the visual Disappears.

in C++ I used :

ComPtr<IDCompositionRectangleClip> clip;

(dcompDevice3->CreateRectangleClip(&clip)

GetWindowRect(hwnd, &hostWindowRect);
clip->SetLeft((float)hostWindowRect.left);
clip->SetRight((float)hostWindowRect.right);
clip->SetTop((float)hostWindowRect.top);
clip->SetBottom((float)hostWindowRect.bottom);
rootVisual->SetClip(clip.Get());

dcompDevice->Commit()

The C# Code I Used:

public IComObject<IDCompositionRectangleClip> clip;

hr = dcompDevice3.Object.CreateRectangleClip(out var c);
clip = new ComObject<IDCompositionRectangleClip>(c);
if (hr != HRESULTS.S_OK)
{
       MessageBox.Show("Failed to Create Rectangle Clip");
}

clip.Object.SetLeft(50.0f);
clip.Object.SetTop(50.0f);
clip.Object.SetRight(500.0f);
clip.Object.SetBottom(500.0f);
visual.Object.SetClip(clip.Object);
dcompDevice3.Object.Commit();

This is how my Window looks like before clipping:

After Clipping the entire visual is missing, and only the window frame is visible:

Edited:

It's not only the RectangleClip not working, i tried SetOffsetX() method of the root Visual it is also having issues, the offset is not set. (in the code below i haven't used the rectangle clip just offset)

Here is the complete code i used to set offset:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Runtime.InteropServices;
using System.Text;
using System.Threading.Tasks;
using System.Windows;
using System.Windows.Controls;
using System.Windows.Data;
using System.Windows.Documents;
using System.Windows.Input;
using System.Windows.Interop;
using System.Windows.Media;
using System.Windows.Media.Imaging;
using System.Windows.Navigation;
using System.Windows.Shapes;
using DirectN;
using MessageBox = System.Windows.MessageBox;

namespace WPF_dllhost
{
    /// <summary>
    /// Interaction logic for MainWindow.xaml
    /// </summary>
    public partial class MainWindow : Window
    {
        [StructLayout(LayoutKind.Sequential)]
        public struct RECT
        {
            public int left, top, right, bottom;
        }

        [StructLayout(LayoutKind.Sequential)]
        public struct DWM_THUMBNAIL_PROPERTIES
        {
            public int dwFlags;
            public RECT rcDestination;
            public RECT rcSource;
            public byte opacity;
            public int fVisible; //BOOL{TRUE,FALSE}
            public int fSourceClientAreaOnly; //BOOL{TRUE,FALSE}
        }

        public IComObject<ID3D11Device> d3ddevice;
        public IComObject<IDXGIDevice> dxgiDevice;
        public IComObject<IDXGIFactory2> dxFactory;
        public IComObject<IDCompositionDesktopDevice> desktopDevice;
        public IComObject<IDCompositionDevice3> dcompDevice3;
        public IComObject<IDCompositionTarget> target;
        public IComObject<IDCompositionVisual2> visual;
        public IComObject<IDCompositionVisual2> rootVisual;
        public IComObject<IDCompositionRectangleClip> clip;

        IntPtr dcomp = IntPtr.Zero;

        [DllImport("dwmapi.dll", CharSet = CharSet.Auto, EntryPoint = "#147")]
        private static extern IntPtr DwmpCreateSharedThumbnailVisual(IntPtr hwndDest, IntPtr hwndSource,uint thumbnailFlags, DWM_THUMBNAIL_PROPERTIES prop,IntPtr device,out IntPtr visualout, out IntPtr thumb);

        [DllImport("user32.dll", SetLastError = true)]
        static extern IntPtr FindWindow(string lpClassName, string lpWindowName);


        public MainWindow()
        {
            InitializeComponent();
        }

        private void CreateCompositionDevice(IntPtr hwnd)
        {
            d3ddevice = D3D11Functions.D3D11CreateDevice(null, D3D_DRIVER_TYPE.D3D_DRIVER_TYPE_HARDWARE, D3D11_CREATE_DEVICE_FLAG.D3D11_CREATE_DEVICE_BGRA_SUPPORT, null, 7);
            if(d3ddevice==null)
            {
                MessageBox.Show("Failed to create D3D11 Device");
            }

            dxgiDevice = new ComObject<IDXGIDevice>(d3ddevice.As<IDXGIDevice>());
            if (dxgiDevice == null)
            {
                MessageBox.Show("Failed to get DXGI Device");
            }

            Functions.DCompositionCreateDevice3(dxgiDevice.Object, new Guid("5f4633fe-1e08-4cb8-8c75-ce24333f5602"), out var pdesktopDevice);
            desktopDevice = new ComObject<IDCompositionDesktopDevice>(Marshal.GetObjectForIUnknown(pdesktopDevice) as IDCompositionDesktopDevice);
            if (desktopDevice == null)
            {
                MessageBox.Show("Failed to Create Composition Desktop Device");
            }

            Guid IID_IDcompositionDevice3 = new Guid("0987CB06-F916-48BF-8D35-CE7641781BD9");
            Marshal.QueryInterface(pdesktopDevice, ref IID_IDcompositionDevice3, out var dcomp3);
            dcompDevice3 = new ComObject<IDCompositionDevice3>(Marshal.GetObjectForIUnknown(dcomp3) as IDCompositionDevice3);
            if (dcompDevice3 == null)
            {
                MessageBox.Show("Failed to Query IDCompositionDevice3 Interface");
            }

            HRESULT hr = desktopDevice.Object.CreateTargetForHwnd(hwnd, true, out var t);
            target = new ComObject<IDCompositionTarget>(t);
            if (hr != HRESULTS.S_OK)
            {
                MessageBox.Show("Failed to Create Composition Target");
            }

            hr = dcompDevice3.Object.CreateVisual(out var v);
            rootVisual = new ComObject<IDCompositionVisual2>(v);
            if (hr != HRESULTS.S_OK)
            {
                MessageBox.Show("Failed to Create Root Visual");
            }

            hr = dcompDevice3.Object.CreateRectangleClip(out var c);
            clip = new ComObject<IDCompositionRectangleClip>(c);
            if (hr != HRESULTS.S_OK)
            {
                MessageBox.Show("Failed to Create Rectangle Clip");
            }


            DWM_THUMBNAIL_PROPERTIES thumbnail=new DWM_THUMBNAIL_PROPERTIES();
            var sourceRect = new RECT
            {
                left =0,
                top = 0,
                bottom = 768,
                right =1366
            };

            var destRect = new RECT
            {
                left = 0,
                top = 0,
                bottom = 768,
                right = 1366
            };


            thumbnail.dwFlags = 0x00000010 | 0x00000008 | 0x00000001 | 0x00000002 | 0x00000004 | 0x4000000;
            thumbnail.opacity = 255;
            thumbnail.fVisible = 1;
            thumbnail.fSourceClientAreaOnly = 1;
            thumbnail.rcDestination = destRect;
            thumbnail.rcSource = sourceRect;

            DwmpCreateSharedThumbnailVisual(hwnd, FindWindow("progman",null), 2, thumbnail, dcomp3, out var desktopVisual, out var thumb);

            visual = new ComObject<IDCompositionVisual2>((IDCompositionVisual2)Marshal.GetObjectForIUnknown(desktopVisual));


            rootVisual.Object.AddVisual(visual.Object, true, null);
            rootVisual.Object.SetOffsetX(200); // not working
            target.Object.SetRoot(rootVisual.Object);
            dcompDevice3.Object.Commit();
        }

        protected override void OnSourceInitialized(EventArgs e)
        {
            base.OnSourceInitialized(e);
            new BorderlessWindow(new WindowInteropHelper(this).EnsureHandle());
            CreateCompositionDevice(new WindowInteropHelper(this).EnsureHandle());
            this.SizeChanged += MainWindow_SizeChanged;
        }

        private void MainWindow_SizeChanged(object sender, SizeChangedEventArgs e)
        {
           

        }
    }
}

Answer 1 · 2021-08-15T14:13:08.000Z

I think it may be due to what's described here: https://stackoverflow.com/questions/61527633/wrong-vtable-generated-by-c-compiler-for-com-object

Many DComp interfaces have members with the same name but with different arguments, for example: IDCompositionRectangleClip's SetLeft has two "overloads".

You can try to redefine them manually (invert members) just to check if it works better, ie:

[ComImport, Guid("9842ad7d-d9cf-4908-aed7-48b51da5e7c2"), InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]
public partial interface IDCompositionRectangleClip : IDCompositionClip
{
    // IDCompositionClip
    
    // IDCompositionRectangleClip
    [PreserveSig]
    HRESULT SetLeft(/* THIS_ _In_ */ IDCompositionAnimation animation);
    
    [PreserveSig]
    HRESULT SetLeft(float left);
    ...
}

Note other notable interop projects have the same issue (but they are not even aware): https://github.com/microsoft/win32metadata

PS: you should use the newer WinRT's Windows.UI.Composition, it's much easier to use in .NET, you won't face this issue, and you don't need DirectN.

Answer 2 · 2021-08-15T14:39:59.000Z

Thank you for helping.