[LINUX][NVIDIA] Doesn't work properly with NVIDIA GPU

Question

[LINUX][NVIDIA] Doesn't work properly with NVIDIA GPU

mercifulboss opened this issue 4 months ago · 21 comments

When using the "CPU" option, the background remover updates as you move around and continues removing the background. When you switch to "GPU - TensorRT" option, it updates the silhouette once and then doesn't update or do anything after the first time.

To Reproduce

Steps to reproduce the behavior:

Go to Background Removal Filter on OBS
Click on "GPU - TensorRT"
See it doesn't update the background

Expected behavior

Supposed to update the background using the "GPU" option

Desktop (please complete the following information):

OS: Arch Linux x86_64
Kernel: 6.9.9-arch1-1
DE: Plasma 6.1.2
CPU: Intel i7-10750H (12) @ 5.000GHz
GPU: NVIDIA GeForce RTX 2060 Mobile
Browser: Firefox
OBS Version: 30.2.0

Answer 1 · 2024-07-18T11:50:47.000Z

can you please share the OBS logs so we can investigate?

Answer 2 · 2024-07-18T17:11:08.000Z

Thank you Roy for the great work!
I have the same issue:

This part of the log is for cpu where it works:

info: [obs-backgroundremoval] get_latest_version:
info: [obs-backgroundremoval] Background filter updated
info: [obs-backgroundremoval] Model models/mediapipe.onnx input 0: name input_1:0 shape (4 dim) 1 x 144 x 256 x 3
info: [obs-backgroundremoval] Model models/mediapipe.onnx output 0: name segment:0 shape (4 dim) 1 x 144 x 256 x 2
info: [obs-backgroundremoval] Allocated 110592 sized float-array for input 0
info: [obs-backgroundremoval] Allocated 73728 sized float-array for output 0
info: [obs-backgroundremoval] Background Removal Filter Options:
info: [obs-backgroundremoval] Source: Background Removal
info: [obs-backgroundremoval] Model: models/mediapipe.onnx
info: [obs-backgroundremoval] Inference Device: cpu
info: [obs-backgroundremoval] Num Threads: 1
info: [obs-backgroundremoval] Enable Threshold: true
info: [obs-backgroundremoval] Threshold: 0.500000
info: [obs-backgroundremoval] Contour Filter: 0.050000
info: [obs-backgroundremoval] Smooth Contour: 0.500000
info: [obs-backgroundremoval] Feather: 0.000000
info: [obs-backgroundremoval] Mask Every X Frames: 1
info: [obs-backgroundremoval] Enable Image Similarity: true
info: [obs-backgroundremoval] Image Similarity Threshold: 35.000000
info: [obs-backgroundremoval] Blur Background: 0
info: [obs-backgroundremoval] Enable Focal Blur: false
info: [obs-backgroundremoval] Blur Focus Point: 0.100000
info: [obs-backgroundremoval] Blur Focus Depth: 0.000000
info: [obs-backgroundremoval] Disabled: true
info: [obs-backgroundremoval] Model file path: /usr/share/obs/obs-plugins/obs-backgroundremoval/models/mediapipe.onnx

This part is for GPU-TensorRT enabled:

info: [obs-backgroundremoval] get_latest_version:
info: [obs-backgroundremoval] get_latest_version:
info: [obs-backgroundremoval] Background filter updated
error: [obs-backgroundremoval] /onnxruntime_src/onnxruntime/core/session/provider_bridge_ort.cc:1209 onnxruntime::Provider& onnxruntime::ProviderLibrary::Get() [ONNXRuntimeError] : 1 : FAIL : Failed to load library libonnxruntime_providers_tensorrt.so with error: libnvinfer.so.8: cannot open shared object file: No such file or directory

error: [obs-backgroundremoval] Failed to create ONNXRuntime session. Error code: 5
info: [obs-backgroundremoval] Background filter updated
info: [obs-backgroundremoval] Background Removal Filter Options:
info: [obs-backgroundremoval] Source: Background Removal
info: [obs-backgroundremoval] Model: models/mediapipe.onnx
info: [obs-backgroundremoval] Inference Device: tensorrt
info: [obs-backgroundremoval] Num Threads: 1
info: [obs-backgroundremoval] Enable Threshold: true
info: [obs-backgroundremoval] Threshold: 0.500000
info: [obs-backgroundremoval] Contour Filter: 0.050000
info: [obs-backgroundremoval] Smooth Contour: 0.500000
info: [obs-backgroundremoval] Feather: 0.000000
info: [obs-backgroundremoval] Mask Every X Frames: 1
info: [obs-backgroundremoval] Enable Image Similarity: true
info: [obs-backgroundremoval] Image Similarity Threshold: 35.000000
info: [obs-backgroundremoval] Blur Background: 0
info: [obs-backgroundremoval] Enable Focal Blur: false
info: [obs-backgroundremoval] Blur Focus Point: 0.100000
info: [obs-backgroundremoval] Blur Focus Depth: 0.000000
info: [obs-backgroundremoval] Disabled: true
info: [obs-backgroundremoval] Model file path: /usr/share/obs/obs-plugins/obs-backgroundremoval/models/mediapipe.onnx
error: [obs-backgroundremoval] Model is not initialized
error: [obs-backgroundremoval] Model is not initialized
error: [obs-backgroundremoval] Model is not initialized

It seems it can not find the right version of libnvinfer (I do have /usr/lib/x86_64-linux-gnu/libnvinfer.so.10)

Answer 3 · 2024-07-23T13:58:56.000Z

Ubuntu 24.04
I have same issue. Part of the log:

error: [obs-backgroundremoval] /onnxruntime_src/onnxruntime/core/session/provider_bridge_ort.cc:1209 onnxruntime::Provider& onnxruntime::ProviderLibrary::Get() [ONNXRuntimeError] : 1 : FAIL : Failed to load library libonnxruntime_providers_tensorrt.so with error: libnvinfer.so.8: cannot open shared object file: No such file or directory

error: [obs-backgroundremoval] Failed to create ONNXRuntime session. Error code: 5
error: [obs-backgroundremoval] Model is not initialized

Answer 4 · 2024-07-23T14:07:11.000Z

What version of CUDA do you have?

Answer 5 · 2024-07-23T14:11:05.000Z

@royshil I suppose many want to use our plugin with CUDA 12. Are you okay with our plugin supporting CUDA 12 and dropping support for CUDA 11? Users who wish to use our plugin with CUDA 11 can still use the older version of our plugin.

Answer 6 · 2024-07-23T18:48:02.000Z

What version of CUDA do you have?

12.2

Answer 7 · 2024-07-23T18:59:26.000Z

CUDA 12 is not supported. Can you try CUDA 11?

Answer 8 · 2024-07-23T21:05:20.000Z

CUDA 12 is not supported. Can you try CUDA 11?

I would try it if I could find a way to downgrade the version

Answer 9 · 2024-07-25T13:15:20.000Z

CUDA 12 is not supported. Can you try CUDA 11?

11 is not supported anymore and no way to downgrade either on Fedora or even Arch it will always give an error

Answer 10 · 2024-08-01T13:56:02.000Z

Same issue here. Even if I were to hack around the issue, it would be onerous to have to switch CUDA versions every time I want to use OBS - and would make it impossible to use both at once.

Answer 11 · 2024-08-03T07:02:06.000Z

Also would love to see support for CUDA 12 for this amazing plugin. Without it, I am forced to use CPU. In order to have acceptable results with CPU it introduces significant CPU consumption which leads to encoder overload and dropped frames.

Took a cursory glance at the codebase but didn't see where the tensorrt relevant code was. Could you point to what makes this break on CUDA 12?

Answer 12 · 2024-08-03T07:06:19.000Z

I don't think nothing would break with CUDA 12. We are just searching for people who can make changes to support CUDA 12 for our code.

Answer 13 · 2024-08-03T07:58:20.000Z

Does tensorrt need to be installed for this to function properly? Or is CUDA theoretically enough (assuming on the supported version). Wonder if that could be the root of the issue as I have the same exact symptoms and only have CUDA 12 installed.

Answer 14 · 2024-08-03T15:41:29.000Z

Does tensorrt need to be installed for this to function properly? Or is CUDA theoretically enough (assuming on the supported version). Wonder if that could be the root of the issue as I have the same exact symptoms and only have CUDA 12 installed.

The problem right now is that it targets the wrong version of the libraries - so it's simply a File Not Found. Since there's a major version change in the library versions, there could be breaking changes. The way I would approach it would be to target the later library version(s) and then test. Others might list all the API calls and form a gap analysis between the library versions, then make targeted fixes if there are any.

Answer 15 · 2024-08-12T18:51:44.000Z

Does disabling "Skip image based on similarity" in the advanced settings fix the issue?

Answer 16 · 2024-08-14T16:07:41.000Z

Does disabling "Skip image based on similarity" in the advanced settings fix the issue?

Doesn't change anything for me.

Running on openSUSE Tumbleweed:

info: [obs-backgroundremoval] Background filter created
info: [obs-backgroundremoval] Background filter updated
error: [obs-backgroundremoval] /onnxruntime_src/onnxruntime/core/session/provider_bridge_ort.cc:1209 onnxruntime::Provider& onnxruntime::ProviderLibrary::Get() [ONNXRuntimeError] : 1 : FAIL : Failed to load library libonnxruntime_providers_tensorrt.so with error: libcudnn.so.8: cannot open shared object file: No such file or directory

error: [obs-backgroundremoval] Failed to create ONNXRuntime session. Error code: 5

Cuda 12.6, TensorRT 10.3.0.26
Even tried symlinking version 9 files to 8 just for the lols, to no avail.. (might have done that incorrectly)

Is there any documentation on what version to use/what to specifically install?
Also, should I downgrade either TensorRT/Cuda (or both) or wait for the fix? 😜

Some more info if you find it useful:

info: Kernel Version: Linux 6.10.3-1-default
info: Flatpak Branch: stable
info: Flatpak Arch: x86_64
info: Flatpak Runtime: runtime/org.kde.Platform/x86_64/6.6
info: App Extensions:
info:  - com.obsproject.Studio.Plugin.BackgroundRemoval
info:  - com.obsproject.Studio.Locale
info: Runtime Extensions:
info:  - org.freedesktop.Platform.GL.default
info:  - org.freedesktop.Platform.GL.nvidia-550-107-02
info:  - org.freedesktop.Platform.openh264
info:  - org.gtk.Gtk3theme.Breeze
info:  - org.kde.Platform.Locale
info:  - org.freedesktop.Platform.GL.default
info: Flatpak Framework Version: 1.15.8
info: Desktop Environment: KDE (KDE)
info: Window System: X11.0, Vendor: The X.Org Foundation, Version: 1.21.1
info: Qt Version: 6.6.3 (runtime), 6.6.3 (compiled)
info: Portable mode: false
info: OBS 30.2.2 (linux)

Answer 17 · 2024-10-07T03:34:10.000Z

Looking forward to a new build with support for CUDA 12