No dynamic scenes were obtained after stage 3 training

[ByChelsea]

Hello, I used the prompt "a dog riding a skateboard“ provided in train.sh, but the result doesn't seem to be 4D; there's hardly any dynamic change. What could be the reason for this?

it70037-test.mp4

[sherwinbahmani]

Hi, is this with the low vram config? This can happen with this setup. Have you tried increasing system.loss.lambda_sds_video as in the readme?

[ByChelsea]

Yes, it is the low vram config. I didn't change the value of system.loss.lambda_sds_video; I kept it at 0.1. I'll try increasing it to see if it yields better results. Thanks!

[sherwinbahmani]

I also found the original implementation to give a bit better results: https://github.com/sherwinbahmani/4dfy

Increasing system.loss.lambda_sds_video towards 1.0 will increase motion significantly

[ByChelsea]

Thanks for the suggestion! I checked and realized that I'm using the original implementation. I'm sorry for the confusion between the two repos when submitting the issues...I'll be careful about this in the future. :)

BTW, it seems that low VRAM (24G) really does significantly reduce the quality...

[sherwinbahmani]

To improve the motion also a better 4D representation will help, like a deformation based representation. The current one with the 4D hash grids was more developed for high quality, but the motion is generally compensated by reducing ystem.loss.lambda_sds_video. So increasing ystem.loss.lambda_sds_video increases motion but also compensates the quality more. In the normal vram setting it seemed to be fine like this, for low vram it might need more hyperparameter tuning.

[ByChelsea]

Thank you very much for the share and suggestions, they are very helpful!