keijiro/NNCam

Renders extremely slow and at a small resolution.

sam598 opened this issue · 7 comments

sam598 commented

The foreground image updates at around 1fps (editor FPS is about 10-20). Tested all 3 models and both architectures with same results. Manually setting the worker's device to CPU, GPU or compute does not change performance.

Additionally the resolution of the foreground image appears extremely small. This could be because _webcam.width and _webcam.height always return a value of 16.

  • Unity 2020.2.1f1
  • Barracuda 1.12.1-preview
  • Macbook Pro 16" 2019
sam598 commented

It seems to run okay on iOS, although there is jitter due to frames arriving out of sync.

sam598 commented

Also adding this at the top of the Controller's update function:

if (!_webcam.didUpdateThisFrame)
    return;

seems to help performance and frame jitter. The update loop may often be faster than a webcam's frame rate, causing the application to run bodypix on the same frame twice.

keijiro commented

Thanks for pointing them out.

I realized that there is a platform-dependent behavior difference in WebCamTexture. On Windows, the dimensions of the texture is updated right after calling .Play(). On macOS, the dimensions are unknown until the device gets actually active. Before that, it returns 16x16.

I added a workaround that initializes _delay texture with 1920x1080 not depending on the actual WebCamTexture dimensions.

This is not an actual solution because it distorts the input image if the webcam doesn't provide a 16:9 wide-screen video stream. It should be fixed in future updates.

Also thanks for suggesting .didUpdateThisFrame. I simply added it.

sam598 commented

It does still run at about 0.5-1fps on a Macbook Pro. Maybe it's a barracuda inference issue?

keijiro commented

How about other lighter models? I added some more models to the release. My M1 MacBook Air runs MobileNetV1-x075-stride16.onnx at 60 fps.

If you're using MBP with Radeon, I'd also recommend changing Device To Use (Preferences -> General -> Device To Use).

sam598 commented

Found the issue. For some reason the webcam itself is only updating once every second (this is using the Macbook Pro's built in facetime camera).

Manually requesting the resolution and framerate fixes the issue and it runs great.
_webcam = new WebCamTexture(1280, 720, 30);

Also having bodypix running in Unity through Barracuda is unbelievably amazing. Thank you for doing the hard part and getting it to work so well.

  • ❤️1
keijiro commented

Thanks for additional testing. WebCamTexture on macOS has several issues for some time. That workaround should be implemented in future updates too.