How can I obtain the correct cam_param for the Neuman dataset?

Hi, thanks for sharing your excellent work!

When I try to render the mesh of smplx for Neuman dataset, I found the pre-processed data you provided (in this link ) does not contain the cam_param.

So I try to use the virtual camera generation script (in this script) to generate the camera parameter for rendering. However, the rendering results is incorrect as shown bellow.

By the way, I have also try to render the mesh for the gyeongsik dataset provided in this link, which uses the virtual camera parameter generated by the above script. The rendered result is correct as shown bellow.

I seems that to render smplx mesh, the camera parameters for different dataset (NeuMan dataset and gyeongsik) are generated in different manner. Would you please tell me how to obtain the 'correct' camera parameters for NeuMan dataset?

Thanks a lot!

Hi,

Each dataset could have different camera parameter.
For the NeuMan dataset, you can check it from the dataloader like here and here.

Hi,

Each dataset could have different camera parameter. For the NeuMan dataset, you can check it from the dataloader like here and here.

After using the correct camera parameter, the rendered smplx seems normal.

However, the rendered result (from this script) still fails to meet the performance you provide under the folder 'renders'. The comparison of the rendered results are shown bellow, (upper: you provide in the pre-process data, lower: the rendered results generated by script )

Would you please tell what opration I am missing to obtain the well-aligned rendered results like you provided?

Thanks a lot.

Hi, the smoothing code assumes input SMPL-X parameters are from reasonable frame rates of a video. For example, 30 fps. However, NeuMan dataset is released after being downsampled to 5 fps, so it would be hard to apply any smoothing code. 5 fps is too sparse to apply some temporal smoothing stuff. That is the reason why I didn't release/use smoothed SMPL-X parameters of NeuMan dataset.

Get it! Without conducting the smoothing operation, the rendered results get correct. Thanks a lot!

Hi Gyeongsik @mks0601, a very quick question: have you re-run COLMAP on NeuMan dataset or just taken what the original dataset provides? Somehow I cannot render the SMPL-X to match the images using the cameras provided by NeuMan. Thanks!

I used the original camera parameters without re-running COLMAP. How do you load camera parameters?

I used the original camera parameters without re-running COLMAP. How do you load camera parameters?

Thanks for the prompt response! It's very good to know! I used the same way as you pointed out:

ExAvatar_RELEASE/avatar/data/NeuMan/NeuMan.py

Line 34 in 5ef87f4

# load cameras

I initially saw the mismatch and thought the COLMAP has been rerun. I will double check my visualization code.

sure. you can post minimally implemented your rendering code here if you want.

sure. you can post minimally implemented your rendering code here if you want.

Thanks for offering that! I just compared with your visualization code and it seems like all optimized SMPL-X parameters are in the camera space (rendering with identity matrix for Rt). I used the loaded Rt from COLMAP to render based on the optimized SMPLX- parameters and that's why there is a mismatch. Now I used an identity matrix for Rt and now it matches. Could you please correct me if I don't understand correctly?

You're correct. All optimized smplx parameters are in camera coordinate space. That's the reason why I convert the coordinates to the world coordinates when creating the avatar in here.