Problems when training A-NeRF in zju_mocap dataset
Yzmblog opened this issue · 5 comments
When I trained A-NeRF in zju_mocap not changing dataloader in this repo, it seems that there will be some artifacts on the floor when visualization. Do you meet the problem when you training on zju_mocap? or whether I did something wrong when training?
Hi, thanks for reaching out.
Since ZJU-Mocap has small pose variation, A-NeRF will just learn everything that won't hurt the reconstruction on training images (it's not constrained by surface model).
If you wish to train an A-NeRF with cleaner background, you can try perturb the background, so that it's A-NeRF cannot capture it easily (and therefore, given up on it as we provided background color to it anyway).
Specifically, see this line here:
https://github.com/LemonATsu/A-NeRF/blob/main/core/dataset.py#L270
You can try replacing it with:
bg = np.random.random(bg.shape).astype(np.float32) * (1 - fg)
So that the background is just a bunch of noise, and A-NeRF cannot forcefully remember this without hurting reconstruction.
I never tried this, but this should work in theory.
Hi, Thanks for your reply! And i have try to replacing the background with noise as you said, but the result is still bad, The disp map is as below:
And i try to change the background to pure black and white, it didn't work out too.
Maybe A-NeRF is not suitable for zju_mocap? or there are some other reasons for this?
"Maybe A-NeRF is not suitable for zju_mocap? or there are some other reasons for this?"
A-NeRF is in general not suitable for dataset with small pose diversity, so yes, zju_mocap is not a suitable dataset for A-NeRF. Like we discussed above: A-NeRF is not constrained by the surface. Novel-view synthesis for this kind of dataset should still be fine to some extend (assuming you train it with multi-view).
If you want to try it a little more, set mask_image=True in the config, along with the noise background mentioned previously. Also, we also have this 'mask supervision setting' that we never use in the original paper, see here:
https://github.com/LemonATsu/A-NeRF/blob/main/run_nerf.py#L245
You can set reg_fn=L1 (or other possible choice here: https://github.com/LemonATsu/A-NeRF/blob/main/core/trainer.py#L159),
and tune the mask loss via reg_coef=.... It basically calculates the loss between the mask and the predicted disp map. You can try with reg_coef=0.001 and see how that works, but I never really tried it.
Again, thanks for reaching out! Let me know if you have further questions.
Hi, thanks for your reply again! And I have try these methods, it's better now. Although still have some artifact, but it's much better! The final val disp image is like this:
Thanks for your discussion and quick reply again!
No problem. I am closing this issue now. Feel free to reopen and comment if you want to discuss further.