Cache TensorRT modules
sjuxax opened this issue · 5 comments
Is your feature request related to a problem? Please describe.
Every time I open the OBS with this plugin installed, there's a 60 sec+ delay while the TensorRT modules are compiled.
Describe the solution you'd like
Most applications use a cache to prevent this from occurring on each startup. That would be great.
Describe alternatives you've considered
Waiting a long time each time I start OBS, which is annoying.
do you have any details on precompilation of tensorRT modules?
Both ONNX Runtime and Tensorflow provide mechanisms for caching TensorRT engines.
I don't have a whole lot of other information, unfortunately, other than it's worked well on the onnxruntime applications I've used.
I forked this repository and added CUDA as an option which loads instantly. I could also look into the TensorRT option mentioned by @sjuxax.
@royshil Should I open a pull request with my changes?
I have also made a small change to the build script, allowing me to set the ONNX Runtime version using variables.
EDIT
The latest commit enables the TensorRT cache (placed in ~/.cache/obs-backgroundremoval/tensorrt). I can confirm that using the cache reduced startup time from ~50 seconds to ~4 seconds, in my case.
@krakowski Feel free to open a pull requests! We appreciate any contributions!