RobustVideoMatting
natxopedreira opened this issue · 21 comments
Hello
I moved/ported/coded/play or whatever is called correctly this repo https://github.com/PeterL1n/RobustVideoMatting to use it with you addon.
Is supercool, is interesting to add to examples? It has multiple inputs/outputs and use initial recurrent states.
hey @natxopedreira,
that is indeed another cool example! what network did you use? Is it a frozen graph or saved model?
looking forward for your commit :)
paul
Its a savedModel, i get 24fps on a 1060 GPU processing a 1920x1080 video. I used the mobilenetV3 one, there is also a resnet50.
Note that do not need to set a background frame for model reference as others implemetations.
I uploaded the example to my fork, please comment :)
https://github.com/natxopedreira/ofxTensorFlow2/tree/main/example_robust_video_matting
Tested it on ubuntu 20.04 w/ a GTX 1080. Runs smoothly w/ GPU libs. Runs, but slowly with CPU libs.
One minor suggestion @natxopedreira in the two config.make files you include your path is hard-coded. If you just remove / comment out those lines, it should work fine (see the other example config.make files).
@natxopedreira also, thanks! I've been looking for a good implementation to use on a project and this appears to be perfect!
@bakercp will remove those hard-coded paths, what fps did you get on a 1080?
Im happy that you found this useful!!!
i totally agree with @bakercp!
just found some time to test it and it really runs super smooth..
i have some changes: insert live video and threaded model, but I am already on it!
example will be uploaded soon!
Thanks for your contribution @natxopedreira!!
Maybe next one i will do it so no corrections are need :) i will try it
Im curious about the fps because on original repo claims a super high fps and wondering what card do you need to get that
I noticed it likes a good camera and good light.
example is currently in the develop branch
and @natxopedreira: don't worry about that! I am so glad you r contributing! I only save the models to our cloud, add a few things here and there. The hard part is finding cool SavedModels!
I am not even sure if the threaded model is equally good for user experience. My problem right now is mixing the current video with the slightly delayed output of the threaded model results in a delayed masking.
Maybe I come back to that later...
Yeah, it's the same issue with the Movement model when processing is too slow. That's just how it is and you can save the submitted frame and skip others to simulate the "slowness" without dropping the main thread frame rate. I thought of doing it the last time, but I can try with the masking model then reintegrate into the movenet wrapper.
The main idea is to allow for good main thread performance even on CPU-only systems. You can always run the model without threading when you want to on a GPU set up, but I think it's good to thread for people trying things out on a laptop, for instance.
@bytosaur i did not test it threaded as i run those in my gpu and performance is ok at 1080, i see you set 640x480 in demo.
Just thinking that in the threaded version you can use a struct or something, you sent pixels and get back tensor and processed pxs from "thread" to draw those so mask is not out of sync.....i don't know if im explaining correctly....
I know is not on topic but i have been working also with openvino that use intel CPU to do acceleration and maybe is a better option for people that wants to run those only on cpu (can be used only with opencv), and also worked with mnn for arm devices.
you sent pixels and get back tensor and processed pxs from "thread" to draw those so mask is not out of sync
Yes, this is what I describe and will look into for @bytosaur's updates on the example so far.
Ah ok, perfect!!!
I used that approach previously on a arm project and worked nice.
@natxopedreira on a standard 1080 (not a 1080 ti), I'm getting about 21 fps from your branch / example. It's using your downsample_ratio 0.25 etc.
The cleaned up example on the develop branch has the same 21 fps performance on the standard GTX1080.
Seems very close to mine 1060ti, just wondering how the author of the model claims those high fps
It achieves 4K 76FPS and HD 104FPS on an Nvidia GTX 1080 Ti GPU
I think it comes down to inference speed, not full-video-pipeline speed. If I run they tests
python inference_speed_test.py --model-variant mobilenetv3 --resolution 1920 1080 --downsample-ratio 0.25
I get about 77 it / s.
They note:
Note 3: We only measure tensor throughput. The provided video conversion script in this repo is expected to be much slower, because it does not utilize hardware video encoding/decoding and does not have the tensor transfer done on parallel threads. If you are interested in implementing hardware video encoding/decoding in Python, please refer to PyNvCodec.
Ah, make sense. Thanks
anyone have inferenced it based on openvino?
hey there,
sorry I forgot about the example, but is now also in the main branch :)