Live_Portrait_Monitor

If you find this project useful, please give it a star ❤️❤️

Monitor results

video6105138957194890677.mp4

video6105138957194890678.mp4

video6105138957194890680.mp4

video6105138957194890682.mp4

video6105138957194890681.mp4

Webcam result

My_img.mp4

🔥 Updates

2024/07/10: 🔥 I released the initial version of the inference code for webcam.
2024/07/16: 🔥 I released the initial version of the inference code for monitor.
2024/07/26: 🔥 I released windows package for faster inference speed. Continuous updates, stay tuned!

Introduction

This repo, named Webcam Live Portrait, Author paper LivePortrait: Efficient Portrait Animation with Stitching and Retargeting Control. I am actively updating and improving this repository for research only. If you find any bugs or have suggestions, welcome to raise issues or submit pull requests (PR) 💖.

🔥 Getting Started

1. Clone the code and prepare the environment

git clone https://github.com/Mrkomiljon/Live_Portrait_Monitor.git
cd Live_Portrait_Monitor

# create env using conda
conda create -n LivePortrait python==3.9.18
conda activate LivePortrait
# install dependencies with pip
pip install -r requirements.txt
pip install mss # for using monitor

2. Download pretrained weights

Download pretrained LivePortrait weights and face detection models of InsightFace from Google Drive or Baidu Yun. All weights have packed in one directory 😊. Unzip and place them in ./pretrained_weights ensuring the directory structure is as follows:

pretrained_weights
├── insightface
│   └── models
│       └── buffalo_l
│           ├── 2d106det.onnx
│           └── det_10g.onnx
└── liveportrait
    ├── base_models
    │   ├── appearance_feature_extractor.pth
    │   ├── motion_extractor.pth
    │   ├── spade_generator.pth
    │   └── warping_module.pth
    ├── landmark.onnx
    └── retargeting_models
        └── stitching_retargeting_module.pth

3. Inference 🚀

python inference_org.py

If the script runs successfully, you will get an output mp4 file named animations/s6--d0_concat.mp4. This file includes the following results: driving video, input image, and generated result.

Unrealtime result

My_photo--d6_concat.mp4

Or, you can change the input by specifying the -s and -d arguments come from webcam:

# to use monitor
python inference_monitor.py -s assets/examples/source/MY_photo.jpg 

# or disable pasting back
python inference_monitor.py -s assets/examples/source/s9.jpg -d assets/examples/driving/d0.mp4 --no_flag_pasteback

# to use original code for inference 
python inference_org.py -s assets/examples/source/s9.jpg -d assets/examples/driving/d0.mp4 --no_flag_pasteback

# more options to see
python inference_org.py -h

# For good result, please play with padding (src/crop.py) for driver head.
    y1 = max(0, y1 - 320)                # Upper part of the head
    y2 = min(frame.shape[0], y2 + 550)   # Lower part of the head
    x1 = max(0, x1 - 360)                # Both sides of the head
    x2 = min(frame.shape[1], x2 + 360)   # Both sides of the head

5. Improved inference speed 🚀🚀🚀

IMG_0.MOV

Windows_results.MOV

2024/07/26: Windows integrated package, one-click run without installation, supporting TensorRT. Thanks to author for supporting TensorRT.
[Optional] If you have already installed CUDA and cuDNN on your Windows computer, please skip this step. I have only verified this on CUDA 12.4. If you haven't installed CUDA or if you encounter CUDA-related errors, you need to follow - these steps for installation::
Download cuda12.2, double-click the exe and follow the default settings to install.
Download the cudnn zip file, extract it and copy the lib, bin, and include folders from the cudnn folder to the CUDA12.4 folder (default is C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.4)
[Optional] Install ffmpeg
Download Live_Portrait_Monitor zip file and extract it.
Enter FasterLivePortrait_High_FPS
For real-time Monitor operation, double-click Monitor.bat. If you want to change the target image, run in command line: camera.bat assets/examples/source/s1.jpg
Note!!! There should be a driver face video on Monitor[1] or [2].
I have adapted the code for readability on monitors and introduced some functionalities. You can easily use it.

Acknowledgements

I would like to thank the contributors of FOMM, Open Facevid2vid, SPADE, InsightFace repositories, for their open research and main authors.

（Some demo images/videos above are sourced from image websites/repos. If there is any infringement, I will immediately remove them and apologize.）