Can you animate any FLAME face model like voca does

Can I use the flame face model outside the vocaset(Training set, Validation set and test set) as a template to generate face animation, how do I set it up, and can I do it using a pre-trained model? Or need to retrain the model

You can check the templates.pkl file and define your own templates.pkl to use new templates. Sure you can use the pre-trained model.

Thanks for your reply! I will try it agin

You can check the templates.pkl file and define your own templates.pkl to use new templates. Sure you can use the pre-trained model.

Hi, do you know how to get a templates.pkl from a single portrait image? Any open-source repo to be found?

@LeoniusChen Hi, DECA is a widely used 3D face reconstruction method by estimating FLAME parameters, and then they can be converted to mesh representations. Please check the FLAME repo for more info.

@LeoniusChen Hi, DECA is a widely used 3D face reconstruction method by estimating FLAME parameters, and then they can be converted to mesh representations. Please check the FLAME repo for more info.

I successfully obtained the template using DECA and FLAME, and replace the template in the codes

CodeTalker/main/demo.py

Line 179 in 1a65bfe

template_file = os.path.join(cfg.data_root, "FLAME_sample.ply")

and

CodeTalker/main/demo.py

Line 136 in 1a65bfe

template_file = os.path.join(cfg.data_root, cfg.template_file)

. However, the result is not satisfying. I wonder if the subject for inference should be carefully chosen?

zh_en_12s_FaceTalk_170809_00138_TA_condition_FaceTalk_170725_00137_TA_audio.mp4

Hi, 'subject' is used to change the template for displaying, since you have utilized your own subject, there should be no effect when you are changing 'subject' in demo.yaml. But I think chaning 'condition' in the config may influence the results, you can have a try. BTW, the unsatisfactory results may be caused by domain gap between Chinese and English. You may check if the English audio can work well.

@Doubiiu @LeoniusChen
Is the color and context information from a single portrait image can be add to tamplate or Mesh?

Hi @Doubiiu @LeoniusChen I'm using DECA and can't get the .pkl template, I'd like to ask you how exactly did you get the template.pkl?