Question about function of RIFE in motion generation.

Question

Question about function of RIFE in motion generation.

YuyangYin opened this issue a year ago · 2 comments

Hi yanqin,
Congratulations on your recent work on Consistent4D. It's truly remarkable. I'm currently studying the paper.
I'd like to inquire about the function of RIFE in temporal and spatial generation. In my experiments, I selected two birds from your demo to test RIFE's 2D video generation capabilities. I found the results to be somewhat blurry, and in some frames, the birds' mouth were missing. I think that RIFE's 2D video generation is not very powerful. Therefore, I'm asking how to comprehend the role of RIFE in your pipeline.
Furthermore, I noticed that your results are outstanding. So, which part do you think is the most crucial in temporal and spatial generation?
Thanks~

YuyangYin commented a year ago

Thanks!

Answer 1 · 2023-11-09T02:07:56.000Z

Thanks for your attention to our work!

We adopt RIFE model in Practical-RIFE, where the authors continuously improve the design of the RIFE model for real-world application. I notice that they have an update recently and the latest model is specially optimized for anime scenes. So maybe you can try it out.

As for the role of RIFE in our pipeline, I think it helps with the spatial and temporal smoothing of the dynamic nerf. Admittedly, sometimes the interpolated result from RIFE is slightly blurry, but consindering that we only apply this function in 64x64 rendering images, the blurriness is negligible. In other words, we just hope this function provide a constrain to ensure the intermediate renderings do not deviate significantly from adjacent renderings.

I think the exploitation of interpolation-based dynerfs, (e.g., k-planes, hex-planes and tensor4d) is the BASIS for temporal and spatial consistency in this ill-posed task, since temporal interpolation technique adopted in these works is naturally prone to temporal consistency, which alleviates the effect of discrete supervisional signal in time axis. Our other designs can make a substantial impact only after exploiting this consistent representation.

Feel free to ask me if you have any further questions : )