Deep learning have developed rapidly in the past few years, many aspects of it have been explored to increase human life quality. In the entertainment industry, many application like tiktok, facebook and so on have applied this advanced technology in their applications, create thousand of trendings. And with the desire to master these technology, this repo is created to be a great step in my life long learning journey. This repo is about an filter app which you can apply a mask from an png image to your face. Beside the mask filder, i also have hat filter and a mode to make the mask floating. I have trained a model here so you can just download it and paste it to the repo folder. If you want to use your own model, you can train it by yourself! i have a file to help you in this.
I have built many features for the app, some of it can be listed here:
- Show facial keypoints
- Apply hat filter
- Apply mask filter
- Make the filter floating To see more about the features, please read the How to use section.
First you must have python installed in your computer. You can go to this page to download the latest version
After that, clone this git repo to a folder. Depend on you OS, you will need to run different file:
- If you are using Linux-like OS, you must run InstallLinux.sh
- If you are using Window, you must run InstallWindow.bat
Before running any filter, you need to active virtual environment. To active the virtual environment, also depend on your OS, you will need to run:
- With Window:
./env/Scripts/activate.bat- With Linux-like OS:
source ./env/bin/activateTo run hat filter, the most basic command you can run is:
python3 index.py hat-filterThis command will use the LuffyHat.png as an filter image to apply the filter
To use other image, you can add the specific path to your filter image:
python3 index.py hat-filter [path_to_your_iamge]You can also change the hat position in vertical axis, to do it you can add the "distance from the face" value:
python3 index.py hat-filter [path_to_your_iamge] [distance]If the distance is short, the hat will be near your face. If the distance is long, the hat will be far from your face.
To run mask filter, you need to have a mask image. The mask image need to have white or black background (no background is even better). Some time the mask may be broken because the edge is blured. In that case, you need to remove that blur. To do this, you need to run this command:
python3 adjustBackground.py [pixel-range] [path_to_your_mask]Where the pixel-range is the range of color in RGB you want to remove in background. If this range is too small, the background can not unblur, but if this range is too big, the background might invade the edge make the whole image become background. The recommended value is 30, but you can play around with it to feel the color ;)
After we run the above command, a window will appear like this
Double click on the area that you want to be the background, the background area would turn gray.
After you finish, press 'q' to save the mask file. The file would automatically save as 'mask.png'.
Some example of bad pixel-range value:
-
The range is too small. As you can see the border of the image have many tiny white dot that we dont expect. It will make the mask look ugly
-
The range is too big. When the range is too big, the background starts to invade the mask area, make it look ugly too
After removing the background, you need to identify the face points in the mask. The face points in the mask is corresponding to the face points in your face when you use the app. You can draw the points follow the face below, the point you set on the mask would be the correspoinding point on the face.
To assign the points to the mask, you need to run this command:
python3 keypoints_predict.pyA window will appear, it will contain the mask like previous step. double click on the point you want to assign. Remember, you need to assign point-after-point respectively. The number of the point you are going to assign is appeared in the terminal. If you mis-assign a point, you can always re-assign it by press 'd'. If you want to assign all the point from beginnning, you can press 'r'. After you assigned all 68 points, the window will auto close. If you dont want to assign anymore, press 'q'.
Now you have all the requirement. Let just run the filter!
python3 index.py mask-filterIn the previous section, you can apply the points in the mask to the corresponding points in your face. But how can you know the exact points? If you want to know, you can run the command below to show the points in your face:
python3 index.py Or
python3 index.py keypointsIf you want to make any of the filters above floating, you can just add "floating-mask" to the end of the command, for example:
python3 index.py mask-filter floating-maskOr if you want to save the video, you can always do that by adding the term "save-video" to the end:
python3 index.py mask-filter save-videoBeside the filtering features, i also provide a program to train the model to predict the keypoints. To do that you need to download this dataset and put it in the project folder. After that you can run:
python3 train.py [path_to_save_file]
You need to specificly indentify the name of the file in the path. It would require you to have gpu in your computer. After you finished training your model, you can open file keypoints_predict.py and change the "resnet-for-face-points-recognize-state-dict.pth" to the path to your model to use your model.
- Many thanks to this video series for teaching me alot about opencv and delaunay triangulatiton
- Thanks to this paper which helped me alot in understanding Kalman Filter
- Thanks to Nguyen Dang Huynh and Le Vu Minh for helping me in the project
- And also, to the person who brought me here, the one I respect the most, Phan Anh Duc
If you have any idea or find any mistake in the project, please tell me via:
- Email: hoangvanphi2004@gmail.com
- Facebook: fb.com/hoangvanphi2004