Siggraph 2017
Visit our Project Page.
Michael Gharbi Jiawen Chen Jonathan T. Barron Samuel W. Hasinoff Fredo Durand
Maintained by Michael Gharbi (gharbi@mit.edu)
Tested on Python 2.7, Ubuntu 14.04.
This is not an official Google product.
To install the Python dependencies, run:
cd hdrnet
pip install -r requirements.txt
Our network requires a custom Tensorflow operator to "slice" in the bilateral grid. To build it, run:
cd hdrnet
make
To build the benchmarking code, run:
cd benchmark
make
Note that the benchmarking code requires a frozen and optimized model. Use
hdrnet/bin/scripts/optimize_graph.py and hdrnet/bin/freeze.py to produce these.
To build the Android demo, see dedicated section below.
Run the test suite to make sure the BilateralSlice operator works correctly:
cd hdrnet
py.test test
We provide a set of pretrained models. One of these is included in the repo
(see pretrained_models/local_laplacian_sample). To download the rest of them
run:
cd pretrained_models
./download.py
To train a model, run the following command:
./hdrnet/bin/train.py <checkpoint_dir> <path/to_training_data/filelist.txt>
Look at sample_data/identity/ for a typical structure of the training data folder.
You can monitor the training process using Tensorboard:
tensorboard --logdir <checkpoint_dir>
To run a trained model on a novel image (or set of images), use:
./hdrnet/bin/run.py <checkpoint_dir> <path/to_eval_data> <output_dir>
To prepare a model for use on mobile, freeze the graph, and optimize the network:
./hdrnet/bin/freeze_graph.py <checkpoint_dir>
./hdrnet/bin/scripts/optimize_graph.sh <checkpoint_dir>
You will need to change the ${TF_BASE} environment variable in ./hdrnet/bin/scripts/optimize_graph.sh
and compile the necessary tensorflow command line tools for this (automated in the script).
We will add it to this repo soon.
Make sure to use Android-NDK 12b. Version 13 and above cause known issues with Tensorflow for Android. We tested on Android SDK 25.0.1.
Change the path to tensorflow, android-sdk and android-ndk in WORKSPACE to match
your environment.
Change the target_features and generators_args parameters in android/BUILD to
match your device/emulator.
Then build:
bazel build //android:hdrnet_demo
Plug your smartphone in, or launch an emulator, then install the package on device:
bazel mobile-install //android:hdrnet_demo
To add a new operator, freeze and optimize a TF model:
./hdrnet/bin/freeze_graph.py <checkpoint_dir>
./hdrnet/bin/scripts/optimize_graph.sh <checkpoint_dir>
Then add a folder in android/assets containing the optimized_graph.pb file. Also add the *.bin weights
for the guidemap.
Finally, add the name of this folder to the filters_array in android/res/values/arrays.xml.
The rendering shader that implements the slice and apply operation is in android/assets/camera_preview.frag.
-
Camera2 API yields YUV images, we convert to RGB to feed the TF input in
android/jni/convert_image_hl.cxx. The processing could be sped up by training a model on YUV images directly. -
OpenGL float textures are in the range [0,1] so we normalize the affine weights. (see
android/jni/convert_output_hl.cxx). The shader needs to undo this normalization.
- The BilateralSliceApply operation is GPU only at this point. We do not plan to release a CPU implementation.
- The provided pre-trained models were updated from an older version and might slightly differ from the models used for evaluation in the paper.
- The HDR+ pretrained model has a different input format (16-bits linear, custom YUV). It will produce uncanny colors if run on standard RGB images. We will release and updated version.