OpenRTiST utilizes Gabriel, a platform for wearable cognitive assistance applications, to transform the live video from a mobile client into the styles of various artworks. The frames are streamed to a server where the chosen style is applied and the transformed images are returned to the client.
Copyright © 2017-2019 Carnegie Mellon University
This is a developing project.
Unless otherwise stated, all source code and documentation are under the Apache License, Version 2.0. A copy of this license is reproduced in the LICENSE file.
Portions from the following third party sources have been modified and are included in this repository. These portions are noted in the source files and are copyright their respective authors with the licenses listed.
| Project | Modified | License |
|---|---|---|
| pytorch/examples | Yes | BSD |
The pre-built OpenRTiST server Docker image requires a GPU for image processing.
OpenRTiST using PyTorch (including the pre-built image) has been tested on Ubuntu 16.04 LTS (Xenial) using several nVidia GPUs (GTX 960, GTX 1060, GTX 1080 Ti, Tesla K40).
Alternatively, OpenRTiST can also use the Intel® OpenVINO toolkit for accelerated processing on CPU and processor graphics on many Intel processors. We have tested OpenVINO support using Ubuntu 18.04 LTS (Bionic) and OpenVINO release 2018.5 on an Intel® Core™ i7-6770HQ processor. OpenVINO is supported when installed from source only.
The OpenRTiST server can run on CPU alone. See below on installing from source for details.
OpenRTiST supports Android and standalone Python clients. We have tested the Android client on Nexus 6, Samsung Galaxy S7, and Essential PH-1.
sudo -iapt-get update
apt-get upgradeStep 3. Install Docker
Follow the steps in the above Docker documentation or use the following convenience script:
curl -fsSL get.docker.com -o get-docker.sh
sh get-docker.shapt-get install nvidia-384Step 5. Install nvidia-docker
# If you have nvidia-docker 1.0 installed: we need to remove it and all existing GPU containers
docker volume ls -q -f driver=nvidia-docker | xargs -r -I{} -n1 docker ps -q -a -f volume={} | xargs -r docker rm -f
apt-get purge -y nvidia-docker
# Add the package repositories
curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | apt-key add -
curl -s -L https://nvidia.github.io/nvidia-docker/ubuntu16.04/amd64/nvidia-docker.list | tee /etc/apt/sources.list.d/nvidia-docker.list
apt-get update
# Install nvidia-docker2 and reload the Docker daemon configuration
apt-get install -y nvidia-docker2
pkill -SIGHUP dockerd
docker pull cmusatyalab/openrtistnvidia-docker run --privileged --rm -it --env DISPLAY=$DISPLAY --env="QT_X11_NO_MITSHM=1" -v /dev/video0:/dev/video0 -v /tmp/.X11-unix:/tmp/.X11-unix:ro -p 7070:7070 -p 9098:9098 -p 9111:9111 -p 22222:22222 -p 8021:8021 cmusatyalab/openrtist nvidia-docker run --privileged --rm -it --env DISPLAY=$DISPLAY --env="QT_X11_NO_MITSHM=1" -v /dev/video0:/dev/video0 -v /tmp/.X11-unix:/tmp/.X11-unix:ro -p 7070:7070 -p 9098:9098 -p 9111:9111 -p 22222:22222 -p 8021:8021 cmusatyalab/openrtist /bin/bashType ifconfig and note the interface name and ip address inside the docker container (for the below examples, we assume eth0 and 172.17.0.2).
ifconfig
Open tmux and create three windows (CTRL-b c).
tmux
In the first tmux window (CTRL-b 0), navigate to /gabriel/server/bin.
cd /gabriel/server/bin
Execute gabriel-control, specifying the interface inside the docker container with the -n flag
./gabriel-control -n eth0
In the next tmux window(CTRL-b 1), execute gabriel-ucomm, specifying the ip address listed earlier with the -s flag. Be sure to include the port 8021.
cd /gabriel/server/bin
./gabriel-ucomm -s 172.17.0.2:8021
In the next tmux window(CTRL-b 2), navigate to the OpenRTiST application directory. Execute the proxy, specifying the ip address listed earlier with the -s flag. Be sure to include the port 8021.
cd /openrtist/server
./proxy.py -s 172.17.0.2:8021
If you wish to compare between running the server on a cloudlet versus a cloud instance, you can launch the following instance type/image from your Amazon EC2 Dashboard:
Instance Type - p2.xlarge (can be found by filtering under GPU compute instance types)
Image - Deep Learning Base AMI (Ubuntu) - ami-041db87c
Ensure that ports 9000-10000 are open in your security group rules so that traffic to/from the mobile client will pass through to the server.
Once the server is running in AWS, you can follow the steps above to setup the server.
Note : If using vanilla Ubuntu Server 16.04 Image, install the required Nvidia driver and reboot.
wget http://us.download.nvidia.com/tesla/375.51/nvidia-driver-local-repo-ubuntu1604_375.51-1_amd64.deb
sudo dpkg -i nvidia-driver-local-repo-ubuntu1604_375.51-1_amd64.deb
sudo apt-get update
sudo apt-get -y install cuda-drivers
sudo reboot
The docker image for the server component also includes the Python client which can be run on a non-Android machine. The python client requires that a USB webcam is connected to the machine.
Step 1. Install Docker
Follow the steps in the above Docker documentation or use the following convenience script:
curl -fsSL get.docker.com -o get-docker.sh
sh get-docker.sh
docker pull cmusatyalab/openrtist
xhost local:root
docker run --privileged --rm -it --env DISPLAY=$DISPLAY --env="QT_X11_NO_MITSHM=1" -v /dev/video0:/dev/video0 -v /tmp/.X11-unix:/tmp/.X11-unix:ro -p 9098:9098 -p 9111:9111 -p 22222:22222 -p 8021:8021 cmusatyalab/openrtist /bin/bash
cd /openrtist/gabriel-client-openrtist-python
vim.tiny config.py
#Here you can edit the image resolution captured by the camera and the frames per second.
./ui.py <server ip address>
Servers can be added by entering a server name and address and pressing the + sign button. Once a server has been added, pressing the 'Play' button will connect to the OpenRTiST server at that address. Pressing the trash can button will remove the server from the server list.
Once the camera is active, the application will be set to 'Clear Display' by default. This will show the frames of the camera without a particular style. You can use the drop-down menu at the top to select a style to apply. Once selected, the frames from the camera will be sent to the server and the style will be applied. The results will be shipped back to the client and displayed. If the 'Iterate Styles Periodically' option is enabled, the dropdown menu will not be shown, and styles will change automatically based upon the interval defined.
In the main activity, there is a switch where you can toggle whether or not to enable the stereoscopic effect. When enabled, the resultant stylized frames will be split into left- and right-eye channels. This effect is interesting to use with various HUDs, such as Google Cardboard.
When this option is enabled, an interval can be set (5-60 seconds). The styles will automatically be iterated through at the occurrence of this interval.
If the 'Show Video Recording Button' switch is enabled, a camera icon will be shown in the lower left corner of the display. Pressing this button will initiate a screen capture. You will be prompted to allow the capture and after closing this dialog, recording will begin. The screen contents will be recorded until the button is pressed a second time. The icon will turn red to indicate that recording is taking place. Video clips will be stored in Movies/OpenRTiST on the devices SD card. NOTE: If the 'Iterate Styles Periodically' option is enabled, styles will not iterate until the screen capture dialog has been accepted and recording has started.
You can toggle whether or not to use the front-facing camera on the main screen. When this option is enabled, a small rotation icon will be present in the upper right hand corner of this display. This icon can be pressed to rotate the view on devices where the styled frames appear upside down.
The OpenRTiST server can use PyTorch or OpenVINO to execute style transfer networks. Please install PyTorch or OpenVINO (or both).
OpenRTiST has been tested with pytorch versions 0.2.0 and 0.3.1 with CUDA support. As the APIs and layer names have been changed in newer releases, please select an appropriate version from here. NOTE: Even if you do not have a CUDA-capable GPU, you can install pytorch with CUDA support and run OpenRTiST on CPU only.
First install torchvision:
pip install torchvision
This will usually install a new version of pytorch. Uninstall and replace this with the older pytorch version (e.g.):
pip uninstall torch
pip install https://download.pytorch.org/whl/cu75/torch-0.2.0.post3-cp27-cp27m-manylinux1_x86_64.whl
Download the latest OpenVINO release from https://software.intel.com/en-us/openvino-toolkit. Full installation instructions are available at https://software.intel.com/en-us/articles/OpenVINO-Install-Linux.
Be sure to install the Intel® Graphics Compute Runtime for OpenCL™ Driver components (under Optional Steps) to enable the use of the integrated GPU.
We recommend Ubuntu 18.04 for a painless install. We have had success with Ubuntu 18.10 and Ubuntu 16.04, but setupvars.sh may not set up the environment correctly, and on the older distro, a new Linux kernel will be needed.
Note: although OpenVINO lists Python 3.5 as a prerequisite, it supports Python 2.7 as well.
Setup environment variables and paths to use OpenVINO with Python 2.7:
$ source /opt/intel/computer_vision_sdk/bin/setupvars.sh -pyver 2.7
Follow the instructions given in Gabriel repo to install and run the control server and ucomm server. E.g., after installing gabriel, run:
$ cd $HOME/gabriel/server/bin
$ ./gabriel-control -n eth0 &
$ ./gabriel-ucomm -n eth0 -s x.x.x.x &
replacing eth0 with your network nic device (e.g., may be eth1 or eno1), and x.x.x.x with the IP address of the machine.
Add the path to gabriel/server to your PYTHONPATH environment variable, e.g.:
$ export PYTHONPATH=$HOME/gabriel/server/:$PYTHONPATH
Note: Gabriel may require Python 2.7.
Start the server like this:
$ cd <openrtist-repo>/server/
$ ./proxy.py -s x.x.x.x:8021
Autodetect: Loaded OpenVINO
TOKEN SIZE OF OFFLOADING ENGINE: 1
Loading network files:
<openrtist-repo>/server/../models/16v2.xml
<openrtist-repo>/server/../models/cafe_gogh-16.bin
Loading model to the plugin
Loading network files:
<openrtist-repo>/server/../models/16v2.xml
.
.
.
<openrtist-repo>/server/../models/udnie-16.bin
Loading model to the plugin
Loading network files:
<openrtist-repo>/server/../models/16v2.xml
<openrtist-repo>/server/../models/weeping_woman-16.bin
Loading model to the plugin
FINISHED INITIALISATION
Note: With OpenVINO using an integrated GPU, it may take up to a minute to preload all of the style models. This is not the case for OpenVINO on CPU, or with PyTorch. Once initialized, the server is ready for clients at this point.
With either PyTorch or OpenVINO, you can run the server in CPU-only mode by first editing config.py and setting USE_GPU = False. By default, OpenRTiST tries to detect and use OpenVINO, and fails over to PyTorch. To force it to use one system, add a line to config.py, setting USE_OPENVINO = <True / False>. Remove this line to re-enable the auto-detection behavior.
4. Run a python or mobile client using source code at gabriel-client-style-(client_type), or the Android client from the Google Play Store.
To run the python client:
cd <openrtist-repo>/gabriel-client-openrtist-python
./ui.py <server ip address>
You can edit the config.py file to change webcam capture parameters.
The prebuilt Android client from the Google Play Store provides an interface to add a server with a custom IP address.
We use COCO 2014 Train Images as our training dataset
wget http://images.cocodataset.org/zips/train2014.zip
unzip train2014.zip -d coco-data/
cd <openrtist-repo>
python train_style.py --dataset <coco-data> --style-image <style-image> --save-model-dir models/ --epochs 2
To disable flicker-loss which removes flicker for temporal consistency in real-time image stream, set --noise-count 0
Please see the CREDITS file for a list of acknowledgments.