In this project a base video and style reference image is taken. The base video is decomposed into frames by using opencv and each frame is passed through a trained deep convolutional neural network over a dataset to repaint base image based on the style of another style reference image. Then the repainted image spitted out of the neural network is again joined to form the output video using opencv. This is the implementation of Neural Style Transfer from the paper A Neural Algorithm of Artistic Style in Keras 1.0.2.
- Numpy (http://www.numpy.org/)
- Keras (http://keras.io/#installation)
- Scipy (https://www.scipy.org/install.html)
- Theano (http://deeplearning.net/software/theano/install.html#install)
- h5py (http://docs.h5py.org/en/latest/build.html)
- sklearn (http://scikit-learn.org/stable/install.html)
- Pillow (https://pillow.readthedocs.io/en/latest/installation.html)
- CUDA (GPU) (http://docs.nvidia.com/cuda/cuda-getting-started-guide-for-mac-os-x/)
- CUDNN (GPU) (https://developer.nvidia.com/cudnn)
- VGG16 file https://drive.google.com/file/d/0Bz7KyqmuGsilT0J5dmRCM0ROVHc/view?usp=sharing
There are 3 parts to identify when we run the script
- Your base video (to artify)
- Your reference image (the art to learn from)
- Your generated video Run the following comand to generate an image in your chosen style
python aistist.py --base_video_path /path/to/your/video --style_reference_image_path /path/to/your/painting --result_prefix /path/to/generated/file/you/create
Other optional commands are
- --image_size: Size of your output frame
- --content_weight: How much to weigh the content
- --style_weight: How much to weigh the style
- --style_scale: How much to scale the style
- --total_variation_weight: Uniformity of the generated frame
- --num_iter: Nmber of iterations
- --rescale_image: to rescale or not to rescale
- --rescale_method: rescale algorithm
- --maintain_aspect_ratio: to maintain aspect ratio or not
- --content_layer: which layer to focus on for content generation
Credits to Somsubra Majumdar, Siraj Raval for vast majority of code.
Contribution => Used AI_Artist and Neural-Style-Transfer code which was implemented to do neural style transfer of images to work for videos. Used opencv to convert videos to frames which was used by the inherited scipt to do style transfer and then the frame image generated was stiched back to a video using opencv.
Major Learnings => Learnt how transfer learning using VGG16 was used in neutral style transfer. Learnt how opencv could be used in video manipulation.