AutomEditor is an AI based automatic video editing tool that helps video bloggers to remove bloopers automatically. It uses multimodal spatio-temporal blooper recognition and localization approaches. The models were trained in keras and integrate feature fusion techniques from face, body gestures (skelethon), emotions progression, and audio features.
http://www.carlostoxtli.com/AutomEditor/frontend/ (You need to link your own backend)
Install the following software
- ffmpeg
- OpenSMILE
- OpenFace
- python 2 or 3 Place the OpenSMILE and OpenFace directories at the side of this cloned directory.
Run the scripts in the following order
- ./backend/feature_extraction/extract_audio_files.py
- ./backend/feature_extraction/generate_audio_feature.py
- ./backend/feature_extraction/generate_face_feature.py
- ./backend/feature_extraction/generate_face_visual.py
- ./backend/feature_extraction/generate_emotion_feature.py
- ./backend/feature_extraction/generate_body_feature.py
- ./backend/feature_extraction/generate_body_visual.py Copy all the .pkl files to the ./backend/data/ folder
For training individual models just give the specific model
- ./backend/experiment/train.py --model emotion_feature For training fusion models just give the comma separated list of models and add the fusion parameter
- ./backend/experiment/train.py --fusion --model body_fusion,face_fusion,audio_feature,emotion_feature
Run the web server
- ./backend/feature_extraction/server.py Expose the frontend directory to the web, for instance:
- cd ./frontend/
- sudo python -m SimpleSTTPServer 80 Navigate the URL, for instance: http://localhost
This work is based on this paper: https://arxiv.org/pdf/1805.00625.pdf
Slides: https://docs.google.com/presentation/d/1ds8nGurfxUgnizWwI61eLh8OTJ4l4K486LwPe38I0Eo
Publication in progress ...