How to convert captured frames to opencv matrix?

[0x915]

I wanted to replace the old winapi window capture with D3D capture.
I modified the SimpleCapture::OnFrameArrived() function and added the opencv code.

void SimpleCapture::OnFrameArrived(winrt::Direct3D11CaptureFramePool const &sender, winrt::IInspectable const &) {
	
	auto frame = sender.TryGetNextFrame();
	auto swapChainResizedToFrame = TryResizeSwapChain(frame);
	
	winrt::com_ptr<ID3D11Texture2D> backBuffer;
	winrt::check_hresult(
		m_swapChain->GetBuffer(
			0,
			winrt::guid_of<ID3D11Texture2D>(),
			backBuffer.put_void()
		)
	);
	auto surfaceTexture = GetDXGIInterfaceFromObject<ID3D11Texture2D>(frame.Surface());
	// copy surfaceTexture to backBuffer
	m_d3dContext->CopyResource(backBuffer.get(), surfaceTexture.get());
	
       // ====================================================
	cv::Mat cvBuffer;
	cvBuffer.create(
		frame.ContentSize().Height,
		frame.ContentSize().Width,
		CV_8UC4
	);
	cv::directx::convertFromD3D11Texture2D(
		GetDXGIInterfaceFromObject<ID3D11Texture2D>(frame.Surface()).get(),
		cvBuffer
	);
	// ====================================================

	DXGI_PRESENT_PARAMETERS presentParameters{};
	m_swapChain->Present1(1, 0, &presentParameters);
	
	if (swapChainResizedToFrame) m_framePool.Recreate(m_device, m_pixelFormat, 2, m_lastSize);
}

My expected design is:
every time a frame arrives, the external opencv matrix buffer is updated (although the current test code buffer is internal). After locking the buffer, the captured picture can be obtained from other threads.

But the current problem is that once cv::directx::convertFromD3D11Texture2D() is executed, the window will be blank with no image, without any errors, and OnFrameArrived() is still being called. I also tried to replace the first parameter GetDXGIInterfaceFromObject(frame.Surface()).get() with surfaceTexture.get() or backBuffer.get() with the same result.

I have not learned D3D development. It seems that the ID3D11Texture2D pointer here cannot be used twice. So how to convert the captured frame to an opencv matrix?

[robmikh]

Unfortunately, I'm not very familiar with OpenCV or with its synchronization mechanism. My guess is that there's a synchronization issue somewhere. I'd try creating a new texture and copying the frame to it, and then using that new texture with OpenCV. Do not access the frame/texture you get from the Capture API after the Direct3D11CaptureFrame object is released.

Here's a repo that does some rudimentary interop between WGC and OpenCV. I would recommend tackling D3D/OpenCV interop and then add in WGC.

https://github.com/robmikh/WGCOpenCVInterop

Closing this issue since I don't have much experience with OpenCV.

[0x915]

Thanks, WGCOpenCVInterop is available and the 2160x3840 has a capture performance of 16-18ms 55-62fps, but it uses 43% of the GPU (REX3070 Laptop) as well as exhausting one CPU (i7-11800H) thread.