Low framerate when running 2 instances of background removal.
MasoBall opened this issue · 14 comments
Describe the bug
I have 2 cams setup and whenever I use this remove background filter with both of them my OBS framerate cannot stay at 60 because of the frames missed due to rendering lag, even though I have plenty of CPU and GPU headroom. This is without even recording or streaming.
To Reproduce
Steps to reproduce the behavior:
- Have 2 cam sources on OBS.
- Have both cams utilize this background removal filter.
- Note extreme framerate drop through OBS even before recording or streaming.
Expected behavior
To be able to utilize background removal on 2 different cams at the same time without tanking frames.
Log and Crash Report
https://obsproject.com/logs/IPAzBW0fOTRnwAgJ
Desktop (please complete the following information):
- OS: Windows 10 and Windows 11
- Plugin Version: 1.1.5
- OBS Version: 29.1.3
Additional context
I7-9700K, RTX 3060 12GB, 16GB Ram
🤔 this is a tough problem to crack.
Are you using GPU acceleration option? e.g. "GPU (DirectML)"
I think if we had native CUDA support on Windows - it might do the trick and fully utilize your 3060.
@umireon CUDA support on Windows is possible - the problem is the access to CUDA DLLs (cudart and cublas)
maybe we can make a Windows release compiled vs. CUDA and ask users to copy the 2-3 DLLs from their CUDA folder?
@MasoBall Can you post here the screenshot of the properties on the obs-backgroundremoval filter? We need the entire property screen on both cameras to investigate why our plugin has so heavy load.
@royshil We can create two variants of our plugin for Windows: DirectML edition and TensorRT (CUDA) edition. These need to be separate binaries because Microsoft does not provide us with the pre-built library that includes both DirectML support and CUDA support. But, yes, it is technically possible.
I suppose that the Robust Video Matting model is one of the most calculation-heavy ones. @royshil Is this correct?
I recommend you try the MediaPipe model which is default and not so calculation-heavy.
@royshil It would be great to output the value of the properties to the OBS log when the plugin is loaded and the properties of the plugin changed.
@umireon i'm thinking about windows CUDA. it shouldnt be very hard, we already supported it in the past, through the NuGet pre-built binary DLLs
do you want to add another build option to https://github.com/obs-ai/onnxruntime-static-win/blob/main/Build-Windows.ps1 to also build a CUDA version? or is it too dificult to do in a github workflow (e.g. getting a CUDA-enabled build system)
we can try to use the NuGet builds again
sure that's easy enough to do. do you want to open an issue? i can implement it quick
This is quite a straightforward feature and it would not need the tracking issue.
do you want to add another build option to https://github.com/obs-ai/onnxruntime-static-win/blob/main/Build-Windows.ps1 to also build a CUDA version? or is it too dificult to do in a github workflow (e.g. getting a CUDA-enabled build system)
Building CUDA-related things on GitHub Actions hosted runner is nearly impossible. Microsoft builds the CUDA-enabled runtime on Azure DevOps, not on GitHub Actions. I don't think we can build ONNX Runtime with CUDA EP because its build process is very complex.
@umireon so how about we use https://www.nuget.org/packages/Microsoft.ML.OnnxRuntime.gpu prebuilt binaries for CUDA?
We need an new issue to discuss CUDA on Windows 😓
I have no idea if it's from the latest update or not, but when I change one of the cams to MediaPipe while leaving the other at Robust Video Matting I'm now able to get a solid 60fps during my test stream!