/nemo

This is an official GitHub repository for the paper, "NEMO: Enabling Neural-enhanced Video Streaming on Commodity Mobile Devices".

Primary LanguageJava

NEMO (MobiCom'20)

This is an official Github repository for the MobiCom paper "NEMO: Enabling Neural-enhanced Video Streaming on Commodity Mobile Devices". This project is built upon Google libvpx, Android Exoplayer, and Qualcomm SNPE and consists of C/C++/Java/Python.
[Project homepage] [Paper] [Video]

If you use our work for research, please cite it.

@inproceedings{yeo2020nemo,
  title={NEMO: enabling neural-enhanced video streaming on commodity mobile devices},
  author={Yeo, Hyunho and Chong, Chan Ju and Jung, Youngmok and Ye, Juncheol and Han, Dongsu},
  booktitle={Proceedings of the 26th Annual International Conference on Mobile Computing and Networking},
  pages={1--14},
  year={2020}
}

Lastly, NEMO is currently protected under the patent and is retricted to be used for the commercial usage.

Project structure

./nemo
├── video                  # Python: Video downloader/encoder
├── dnn                    # Python: DNN trainer/converter
├── cache_profile          # Python: Anchor point selector
├── player                 # Java, C/C++: Android video player built upon Exoplayer and the SR-integrated codec
./third_party
├── libvpx                 # C/C++: SR-integrated codec

Prerequisites

Guide

We provide a step-by-step guide with a single video (which content is product review).
All the folloiwing commands must be executed inside the docker.

1. Setup

  • Clone the NEMO docker repository
git clone https://github.com/chaos5958/nemo-docker.git
  • Build the docker image
cd ${HOME}/nemo-docker
./build.sh
  • Run & Attach to the docker
cd ${HOME}/nemo-docker
./run.sh
  • Clone the NEMO main repository
git clone --recurse-submodules https://github.com/kaist-ina/nemo.git ${NEMO_CODE_ROOT}
  • Download/Setup the Qualcomm SNPE SDK as follow:
./nemo
├── third_party
    ├── snpe
        ├── benchmarks
        ├── bin
        ├── include
        ├── lib
        ├── models
        ├── share
        ...

2. Prepare videos

  • Download a Youtube video
$NEMO_CODE_ROOT/nemo/tool/script/download_video.sh -c product_review
  • Encode the video
$NEMO_CODE_ROOT/nemo/tool/script/encode_video.sh -c product_review

Details are described in this file.

3. Prepare DNNs

  • Train a DNN
$NEMO_CODE_ROOT/nemo/dnn/script/train_video.sh -g 0 -c product_review -q high -i 240 -o 1080
  • (Optional) Convert the TF model to the Qualcomm SNPE dlc
$NEMO_CODE_ROOT/nemo/dnn/script/convert_tf_to_snpe.sh -g 0 -c product_review -q high -i 240 -o 1080
  • (Optional) Test the dlc on Qualcomm devices
$NEMO_CODE_ROOT/nemo/dnn/script/test_snpe.sh -g 0 -c product_review -q high -r 240 -s 4 -d [device id]

Details are described in this file.

4. Generate a cache profile

  • Setup: Build the SR-integrated codec (x86_64)
$NEMO_CODE_ROOT/nemo/cache_profile/script/setup.sh
  • Generate the cache profile using the codec
$NEMO_CODE_ROOT/nemo/cache_profile/script/select_anchor_points.sh -g 0 -c product_review -q high -i 240 -o 1080 -a nemo
  • (Optional) Analyze frame dependencies & frame types
$NEMO_CODE_ROOT/nemo/cache_profile/script/analyze_video.sh -g 0 -c product_review -q high -i 240 -o 1080 -a nemo

Details are described in this file.

5. Compare NEMO vs. baselines

  • Setup: Build the SR-integrated codec (arm64-v8)
$NEMO_CODE_ROOT/nemo/test/script/setup_local.sh
  • Setup: Copy data to mobile devices
$NEMO_CODE_ROOT/nemo/test/script/setup_device.sh -c product_review -q high -r 240 -a nemo_0.5 -d [device id]
  • Measure the latency
$NEMO_CODE_ROOT/nemo/test/script/measure_latency.sh -c product_review -q high -r 240 -a nemo_0.5 -d [device id]
  • Measure the quality
$NEMO_CODE_ROOT/nemo/test/script/measure_quality.sh -g 0 -c product_review -q high -i 240 -o 1080 -a nemo_0.5

Details are described in this file.

6. Play NEMO in Android smartphones

  • Setup: Copy data to mobile devices
$NEMO_CODE_ROOT/nemo/player/script/setup_device.sh -c product_review -q high -r 240 -a nemo_0.5 -d [device id] -a nemo_0.5_16
  • Run the NEMO player
  1. Enable USB debugging and USB install at mobiles.
  2. Open $NEMO_CODE_ROOT/nemo/player using Android Studio.
  3. Set ndk path in local.properties as ndk.dir=[ndk folder]/android-ndk-14b; I used android-ndk-14b.
  4. Build & run it!
App information
* product review: You can choose content.
* 1: it is deprecated.
* high: You can choose DNN quality level.
* Decode-Cache: You can choose Decode, Decode-SR (=per frame super-resolution), Decode-cache (=NEMO).
* None: You can choose total playback time.
* 0.5: You can choose the cache profile, 0.5 represents the quality threshold compared to per-frame super-resolutionl.