Empty renders on custom data

[angusev]

Hi

First of all, thanks for the great work, very cool results! 🚀

I successfully managed to run your code on your data, however it failed to train on my data. Starting from the first iteration, all renders are empty and all losses except mask_loss are equal to zero. Could you please tell me which point I should pay attention to?

[zhengyuf]

Hey,
I would say the problem is likely the camera matrix. The first iteration should show a ball. What’s your camera matrix? What are the ray directions and camera locations?

[angusev]

The only camera parameters provided by me are intrinsic matrix numbers. My video resolution is 2560x1920, so I added to the prepocess.sh script:

fx=1748.958740234375
fy=1746.06689453125
cx=965.0192260742188
cy=1283.6842041015625

If I understand correctly, the extrinsic parameters should be calculated using an approximation of flame. By the end of the preprocessing, the flame_params.json file looks like this:

{"frames": 
  [{
    "file_path": "./image/1", 
    "world_mat": [
      [1.0, 0.0, 0.0, -1.2967017889022827], 
      [0.0, 1.0, 0.0, 1.9440617561340332], 
      [0.0, 0.0, 1.0, -3.0692138671875]
     ], 
    "expression": ...

Rotation is an identity and it looks ok, but for some reason the translation is quite large.

[zhengyuf]

Oh, one problem is that, the fx, fy, cx and cy need to be adjusted to be the ones after cropping and resizing, the code doesn't automatically do that. So cx and cy should be around 256.0 for example.
You can either calculate by hand, or crop and resize your calibration images in the same way and run the calibration again.

[angusev]

@zhengyuf, thanks for your assistance, this helped!)