Reference image in training

[rakesh-reddy95]

@HeliosZhao Why the reference image is been used in training and does it make any significant difference if I use a different masked_reference image during the inference? If that doesn't make any difference then what is the use of reference image in training?

[HeliosZhao]

Hi @rakesh-reddy95 ,

Sorry for the confusion. At each training iteration, we randomly select one frame from the source video and view it as the reference image (Fig. 3), instead of ``masked_reference image'', which aims at maintaining the ability to use visual information.

Therefore, you can use any other reference image during inference to change the protagonist.

[rakesh-reddy95]

Thank you.