Live demo: http://crfasrnn.torr.vision
Caffe version: http://github.com/torrvision/crfasrnn
This repository contains the codes implementing the following papers:
Keras/Tensorflow code for the "CRF-RNN" semantic image segmentation method, published in the ICCV 2015 paper Conditional Random Fields as Recurrent Neural Networks. This paper was initially described in an arXiv tech report. The online demo of this project won the Best Demo Prize at ICCV 2015. Original Caffe-based code of this project can be found here. Results produced with this Keras/Tensorflow code are almost identical to that with the Caffe-based version.
If you use this code/model for your research, please cite the following paper:
@inproceedings{crfasrnn_ICCV2015,
author = {Shuai Zheng and Sadeep Jayasumana and Bernardino Romera-Paredes and Vibhav Vineet and
Zhizhong Su and Dalong Du and Chang Huang and Philip H. S. Torr},
title = {Conditional Random Fields as Recurrent Neural Networks},
booktitle = {International Conference on Computer Vision (ICCV)},
year = {2015}
}
The above code is modified to incorporate papers published in ICIP 2018, Superpixel-enhanced Pairwise Conditional Random Fields for Semantic Segmentation. The modified code is capable of incorporating superpixel-cues in the CRFs to further increase the accuracy simply by feeding in one or multiple filtered images in the input.
If you choose to add superpixels by using this code/model for your research, please cite the following paper:
@INPROCEEDINGS{8451218,
author={L. Sulimowicz and I. Ahmad and A. Aved},
booktitle={2018 25th IEEE International Conference on Image Processing (ICIP)},
title={Superpixel-Enhanced Pairwise Conditional Random Field for Semantic Segmentation},
year={2018},
volume={},
number={},
pages={271-275},
month={Oct},}
$ git clone https://github.com/sadeepj/crfasrnn_keras.git
The root directory of the clone will be referred to as crfasrnn_keras
hereafter.
Note: If you are using a Python virtualenv, make sure it is activated before running each command in this guide.
Use the requirements.txt
file (or requirements_gpu.txt
, if you have a GPU device) in this repository to install all the dependencies via pip
:
$ cd crfasrnn_keras
$ pip install -r requirements.txt # If you have a GPU device, use requirements_gpu.txt instead
As you can notice from the contents of requirements.txt
, we only depend on tensorflow
, keras
, and h5py
. Additionally, Pillow
is required for running the demo.
After installing the dependencies, run the following commands to make sure they are properly installed:
$ python
>>> import tensorflow
>>> import keras
You should not see any errors while importing tensorflow
and keras
above.
Run make
inside the crfasrnn_keras/src/cpp
directory:
$ cd crfasrnn_keras/src/cpp
$ make
Note that the python
command in the console should refer to the Python interpreter associated with your Tensorflow installation before running the make
command above.
You will get a new file named high_dim_filter.so
from this build. If it fails, refer to the official Tensorflow guide for building a custom op for help.
Note: This make script works on Linux and macOS, but not on Windows OS. If you are on Windows, please check this issue and the comments therein for build instructions. The official Tensorflow guide for building a custom op does not yet include build instructions for Windows.
Download the model weights from here or here and place it in the crfasrnn_keras
directory with the file name crfrnn_keras_model.h5
.
$ cd crfasrnn_keras
$ python run_demo.py
If all goes well, you will see the segmentation results in a file named "labels.png".
Any extra set of pre-processed segmented image needs to be firstly uploaded:
seg_data, seg_h, seg_w = util.get_preprocessed_image(segment_file)
Together with img_data, we feed data into the system as a list [img_data, seg_data]. If we want to feed in more than one set of segmented images, simply put them in the list.
probs = model.predict([img_data, seg_data], verbose=False)[0, :, :, :]
The following images show the difference w/o CRF.
No CRF With Pairwise CRF With superpixel
- Current implementation of the CrfRnnLayer only supports batch_size == 1
- An experimental GPU version of the CrfRnnLayer that has been tested on CUDA 9 and Tensorflow 1.7 only, is available under the
gpu_support
branch. This code was contributed by thwjoy.