Skybox follows the camera

[dimtass]

Hi. First, thanks for providing this in github for us to experiment.

I've tried to use a custom video, but it seems like the skybox is always following the camera.

This is an example:
https://youtu.be/Ogt2GM-H2LU

And this is my json file:

{
  "net_G": "coord_resnet50",
  "ckptdir": "./checkpoints_G_coord_resnet50",

  "input_mode": "video",
  "datadir": "./test_videos/test.mp4",
  "skybox": "district9ship.jpg",

  "in_size_w": 384,
  "in_size_h": 384,
  "out_size_w": 1280,
  "out_size_h": 720,

  "skybox_center_crop": 0.6,
  "auto_light_matching": false,
  "relighting_factor": 0.9,
  "recoloring_factor": 1.0,
  "halo_effect": true,

  "output_dir": "./jpg_output",
  "save_jpgs": true
}

The file is 1280x720.

Any idea why this happens?

Regards.

[jiupinjia]

Hi @dimtass, I have checked your input videos, and I believe the problem is in the sky background. As I mentioned in my preprint paper Sec 4.4, one of the limitations of the method is that it only works with input videos with rich sky textures. When there are no textures, the motion of the virtual camera cannot be accurately modeled. To better understand this, I can give you an example. In the figure below, the left one shows a frame with rich sky textures but the right one does not.

I have put SkyAR version 2.0 on my agenda, where I will be working on a more robust motion estimation (motion propagation from foreground to background) and hopefully, the SkyAR 2.0 will have no more requirements on the sky textures. Thanks again for your valuable feedback.

[dimtass]

Hi @jiupinjia thanks for the answer. I thought that might be the case, thanks for confirming.
Yes it's a bit hard to lock to a grey sky.

I've also tried to apply a high pass filter to the exported jpeg inputs and even though the original source is 1080p there is no much -if at all- information in the sky to lock. Also jpeg compression doesn't help much.

High-pass

It seems my input was the worst case, tree tops and grey cloudy sky.

Again, thanks for the prompt reply.

[jiupinjia]

Yes, you're right. But no worry, we are already working on such cases these days. Hopefully, a v2.0 will soon be available here and this problem will be nicely solved.