Official GitHub repository for FSRT: Facial Scene Representation Transformer for Face Reenactment from Factorized Appearance, Head-pose, and Facial Expression Features (accepted to CVPR 2024) by Andre Rochow, Max Schwarz, and Sven Behnke.
[Paper] [Project Page]
Training code will be released soon!
The animated sequences (bottom row) are generated by transferring the motion extracted from the driving video (left) to the person in the source image (top row).
The source images and driving videos are selected from the official VoxCeleb test set. We demonstrate both animation modes, absolute and relative motion transfer, supported by our method.
vox_abs.mp4
vox_rel.mp4
Our model generalizes to source images from the CelebA-HQ dataset and driving videos from the official VoxCeleb2 test set.
CelebA-HQ_rel_1.mp4
CelebA-HQ_rel_2.mp4
Please complete the following steps.
Clone the repository:
git clone https://github.com/andrerochow/fsrt.git
cd fsrt
We recommend to create a new conda environment:
conda create -n fsrt python=3.9
conda activate fsrt
This code requires at least Python 3.9 and PyTorch.
-
Install PyTorch (>= 1.12.0)
-
Additional dependencies can be installed via:
pip install -r requirements.txt
-
In case you want to animate with relative motion transfer and automatically find a best-matching frame, you need to install the face-alignment library:
git clone https://github.com/1adrianb/face-alignment cd face-alignment pip install -r requirements.txt python setup.py install
Pretrained models can be found at google-drive.
The keypoint detector weights should be located at fsrt_checkpoints/kp_detector.pt
. Note that all pretrained checkpoints are trained using the same keypoint detector weights.
python demo.py --checkpoint fsrt_checkpoints/vox256.pt --config runs/vox256/vox256.yaml --source_image path/to/source --driving_video path/to/driving --source_idx 0 --relative --adapt_scale --find_best_frame
python demo.py --checkpoint fsrt_checkpoints/vox256.pt --config runs/vox256/vox256.yaml --source_image path/to/source --driving_video path/to/driving --source_idx 0
Our FSRT implementation ist based on the PyTorch implementation of Scene Representation Transformer and First Order Motion Model for Image Animation.
@inproceedings{rochow2024fsrt,
title={{FSRT}: Facial Scene Representation Transformer for Face Reenactment from Factorized Appearance, Head-pose, and Facial Expression Features},
author={Rochow, Andre and Schwarz, Max and Behnke, Sven},
booktitle={IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
year={2024}
}