/Remote-Sensing-Image-Classification

Remote sensing image classification based on deep learning

Primary LanguagePythonMIT LicenseMIT

Remote sensing image classification

This project focuses on remote sensing image classification using deep learning.

The current implementations are based on PyTorch and Keras with TensorFlow backend.

Feel free to contact me if you need any further information: liushengjie0756 AT gmail.com

Overview

In the script, we first conduct image segmentation and divide the image to several objects. Then, we generate training samples and train a network. The network is used to predict the whole image. Finally, the object-based post-classification refinement strategy is utilized to refine the classification maps.

Networks

  • Wide Contextual Residual Network - WCRN [2]
  • Double Branch Multi Attention Mechanism Network - DBMA [3]
  • Residual Network with Average Pooling - ResNet99_avg
  • Residual Network - ResNet99 [4]
  • Deep Contextual CNN - DCCNN [5]

Requirements

pytorch==1.1.0 # for PyTorch implementation
skimage==0.15.0
sciPy==1.0.0
sklearn==0.19.1
keras==2.1.6 # for Keras implementation
tensorflow==1.9.0 # for Keras implementation

Data sets

You can download the hyperspectral data sets in matlab format at: http://www.ehu.eus/ccwintco/index.php/Hyperspectral_Remote_Sensing_Scenes

Then, you can convert the data sets to numpy.array

How to use

Run demo_pytorch.py

You will see two predicted maps under the current directory when finished. One is raw classification, and the other is after object-based post-classification refinement (superpixel-based regularization).

This implementation is based on PyTorch using the Wide Contextual Residual Network [2].

Run demo_keras.py

This implementation is based on Keras with TensorFlow backend.

For this demo, the dafault network is DBMA. By changing the parameter - patch, which controls the window size of each sample, other networks will be applied.

Separate training and testing

Some imagery may be too large to be loaded in memory at once. For this scenario, we use subsets of the imagery, and separate the training and testing parts so that all the samples can be used for training. To do so, you need to decide how to clip the imagery and fill in the arguments in <demo_keras_loadsamples.py>. The workflow of separate training and testing goes as follows.

  • First, run <demo_keras_loadsamples.py> to generate training samples and save them under current dir.
  • Then, run <demo_keras_train.py> to train the model, and the model will be saved under current dir.
  • Finally, run <demo_keras_predict.py> to predict the whole image.

Patch and the corresponding network

  • patch==5: WCRN
  • patch==7: DBMA
  • patch==9: ResNet99_avg

Networks' performance

Keras

Network WCRN DBMA ResNet99 ResNet99_avg DCCNN
train time (s) 18 222 21 20 41
test time (s) 12 199 22 21 18
OA (%) 83.00 86.86 72.34 86.68 77.54

The experiments are based on Keras with TensorFlow backend using 10 samples per class with augmentation, conducted on a machine equipped with Intel i5-8400, GTX1050Ti 4G and 8G RAM. The OA is of raw classification averaged from 10 Monte Carlo runs.

Network WCRN DBMA ResNet99 ResNet99_avg DCCNN
train time (s) 9 77 11 10 18
test time (s) 13 133 22 19 16
OA (%) 72.77 74.93 62.47 74.50 65.51

The experiments are based on Keras with TensorFlow backend using 5 samples per class with augmentation, conducted on a machine equipped with Intel i5-8500, GTX1060 5G and 32G RAM. The OA is of raw classification averaged from 10 Monte Carlo runs.

Network WCRN DBMA ResNet99 ResNet99_avg DCCNN
train time (s) 91 755 98 88 132
test time (s) 14 132 22 20 17
OA (%) 77.91 81.14 78.22 79.56 77.05

The experiments are based on Keras with TensorFlow backend using 5 samples per class with augmentation and pseudo samples, conducted on a machine equipped with Intel i5-8500, GTX1060 5G and 32G RAM. The OA is of raw classification averaged from 10 Monte Carlo runs.

Baseline for data sets
Data WCRN DBMA ResNet99 ResNet99_avg DCCNN
Pavia University - - - - -
Pavia Center - - - - -
Indian Pine - - - 80.21 (2.49) -
Salinas Valley - - - - -
KSC - - - 95.08 (0.96) -
University of Houston - - - - -
Flevoland 77.25 (1.84) 77.29 (2.55) - 81.66 (1.01) -
Foulum 95.87 (1.06) 97.99 (1.20) - 98.26 (1.11) -

The experiments are based on Keras with TensorFlow backend using 10 samples per class, conducted on a machine equipped with Intel i5-8500, GTX1060 5G and 32G RAM. The OA is of raw classification averaged from 10 Monte Carlo runs.

Pytorch

Network WCRN WCRM-group WCRN-normal WCRN-bn-normal wcrn-bn-default resnet99-torch
train time (s) 17 49 17
test time (s) 13 20 13
OA (%) 79.88 82.16 78.83 80.36 85.25

The experiments are based on Pytorch backend using 10 samples per class with augmentation, conducted on a machine equipped with Intel i7-8700 and 32G RAM (no cuda). The OA is of raw classification averaged from 10 Monte Carlo runs.

To do

  • Add PyTorch implementation of DBMA and ResNet99_avg
  • Active learning
  • Multitask deep learning

References

[1] Liu, S., Qi, Z., Li, X. and Yeh, A.G.O., 2019. Integration of Convolutional Neural Networks and Object-Based Post-Classification Refinement for Land Use and Land Cover Mapping with Optical and SAR Data. Remote Sens., 11(6), p.690.

[2] Liu, S., Luo, H., Tu, Y., He, Z. and Li, J., 2018, July. Wide Contextual Residual Network with Active Learning for Remote Sensing Image Classification. In IGARSS 2018, pp. 7145-7148.

[3] Ma, W.; Yang, Q.; Wu, Y.; Zhao, W.; Zhang, X. Double-Branch Multi-Attention Mechanism Network for Hyperspectral Image Classification. Remote Sens. 2019, 11, 1307.

[4] Liu, S., and Shi, Q., 2019. Multitask Deep Learning with Spectral Knowledge for Hyperspectral Image Classification. arXiv preprint arXiv:1905.04535.

[5] Lee H. Lee and H. Kwon, "Going Deeper With Contextual CNN for Hyperspectral Image Classification," in IEEE Transactions on Image Processing, vol. 26, no. 10, pp. 4843-4855, Oct. 2017.