How do you fit the expression components to the 52 blendshapes defined by Apple?
Closed this issue · 12 comments
Fitstly, many thanks for your excellent work!
I wonder how you fit the expression components to the 52 blendshapes defined by Apple, can you share the way to do that?
Thanks!
Actually, this process contains lot of manual steps:
(1) Fit the FaceVerse model to the Apple's neutral model by Non-rigid ICP + some manual adjustments using Blender;
(2) Find the corresponding points and use these points and the 52 pre-defined Apple's blendshapes to generate 52 FaceVerse blendshapes by non-rigid ICP;
(3) Manually adjust the generated 52 FaceVerse blendshapes using Blender (fix some bugs and smooth some rugged regions).
@LizhenWangT does these fixed 52 blendshapes avaiable? in fbx or obj format?
@LizhenWangT does these fixed 52 blendshapes avaiable? in fbx or obj format?
In FaceVerse version2, just set the expression parameters to 1, you can get the vertices of 52 blendshapes. A simple example like:
model_path = 'facemodel/faceverse_simple_v2.npy'
model_dict = np.load(model_path, allow_pickle=True).item()
# print the name of 52 blendshapes
print(model_dict['exp_name_list'])
meanshape = model_dict['meanshape'].reshape(-1, 3)
tri = model_dict['tri']
meantex = model_dict['tri'].reshape(-1, 3)
expBase = model_dict['exBase']
blendshape_0_ver = meanshape + expBase[:, 0].reshape(-1, 3)
# save the model as ply format (or any other format)
from util_functions import ply_from_array_color
ply_from_array_color(blendshape_0_ver, meantex.astype(np.uint8), tri, 'blendshape_0.ply')
@LizhenWangT > Actually, this process contains lot of manual steps: (1) Fit the FaceVerse model to the Apple's neutral model by Non-rigid ICP + some manual adjustments using Blender; (2) Find the corresponding points and use these points and the 52 pre-defined Apple's blendshapes to generate 52 FaceVerse blendshapes by non-rigid ICP; (3) Manually adjust the generated 52 FaceVerse blendshapes using Blender (fix some bugs and smooth some rugged regions).
Thanks for your quick reply!
But I not really understand the second step, would you mind describe it in a little detail? After the first step, you have the corresponding points in neutral expression between your model and arikt, but how to generate the orther 51 bs by non-rigid icp?And can you give me some reference paper or software?
Best,
@hungsing92 In step 2, we first pull the FaceVerse corresponding points to the position of arkit models. Since our point distribution is denser than apple's, only a small part of the points (which indicates the corresponding points) is pulled. Then we fix these pulled points, pull other points according to the loss function provided by non-rigid ICP. As it is a quite simple step, I think some easier smooth terms also work for this task.
Thanks for your great work. How to download the 52 blendshapes defined by Apple?
Actually, this process contains lot of manual steps: (1) Fit the FaceVerse model to the Apple's neutral model by Non-rigid ICP + some manual adjustments using Blender; (2) Find the corresponding points and use these points and the 52 pre-defined Apple's blendshapes to generate 52 FaceVerse blendshapes by non-rigid ICP; (3) Manually adjust the generated 52 FaceVerse blendshapes using Blender (fix some bugs and smooth some rugged regions).
do you mean you manually adjust and create 52 meshes of FaceVerse model with respect to each 52 arkit blendshape meshes in Blender? And then the 52 FaceVerse meshed are loaded as the expression base?
second question: Are id base acquired by PCA and the expression base acquired by the aforementioned pipeline?
I got a question. Since you adjust the dataset, how to map the origin FaceVerse parameter space to the newly introduced 52 blendshapes param space by Apple ?
Do you need to train the model again using the new dataset or there's some way else easier?
@hungsing92 In step 2, we first pull the FaceVerse corresponding points to the position of arkit models. Since our point distribution is denser than apple's, only a small part of the points (which indicates the corresponding points) is pulled. Then we fix these pulled points, pull other points according to the loss function provided by non-rigid ICP. As it is a quite simple step, I think some easier smooth terms also work for this task.
what is mean of “pull“?