Graph-in-Graph Convolutional Network for Hyperspectral Image Classification (TNNLS 2022)

Sen Jia, Shuguo Jiang, Shuyu Zhang, Meng Xu, Xiuping Jia

The first two authors contribute equally to this work

Abstract: With the development of hyperspectral sensors, accessible hyperspectral images (HSIs) are increasing, and pixel-oriented classification has attracted much attention. Recently, graph convolutional networks (GCN) have been proposed to process graph-structured data in non-euclidean domains, and have been employed in HSI classification. But most methods based on GCN are hard to sufficiently exploit information of ground objects due to feature aggregation. To solve this issue, in this paper, we proposed a graph-in-graph (GiG) model and a related GiG convolutional network (GiGCN) for HSI classification from superpixel viewpoint. The graph-in-graph representation covers information inside and outside superpixels, respectively corresponding to the local and global characteristics of ground objects. Concretely, after segmenting HSI into disjoint superpixels, each one is converted to an internal graph. Meanwhile, an external graph is constructed according to the spatial adjacent relationships among superpixels. Significantly, each node in the external graph embeds a corresponding internal graph, forming the so-called graph-in-graph structure. Then, GiGCN composed of internal and external graph convolution is designed to extract hierarchical features and integrate them into multiple scales, improving the discriminability of GiGCN. Ensemble learning is incorporated to further boost the robustness of GiGCN. It is worth noting that we are the first to propose graph-in-graph framework from superpixel point and the GiGCN scheme for HSI classification. Experiment results on four benchmark data sets demonstrate that our proposed method is effective and feasible for HSI classification with limited labeled samples. For study replication, the code developed for this study is available at https://github.com/ShuGuoJ/GiGCN.git.

Network Architecture

This is our ensemble flowchart. With the ensmeble learning, GiGCN is more robust in various scenes.

Comparison with State-of-the-art Methods

In comparative experiments, GiGCN is compared with other eight state-of-the-art methods for hyperspectral image classification. Ten samples for each class is chosen for training models and the others are used to test. To alleviate biases, the above operation is run ten times.

	PaviaU			Salinas			GF5			YC
	OA (%)	AA (%)	kappa	OA (%)	AA (%)	kappa	OA (%)	AA (%)	kappa	OA (%)	AA (%)	kappa
Two-CNN	78.20	76.02	0.71	76.37	85.25	0.74	87.08	82.37	0.85	85.20	81.18	0.82
3DVSCNN	76.43	75.47	0.69	89.17	94.07	0.88	84.99	79.02	0.83	77.33	75.14	0.73
HSGAN	72.17	74.66	0.64	83.44	89.67	0.82	88.84	81.50	0.87	92.74	88.46	0.91
SSLstm	69.59	72.77	0.62	81.20	87.04	0.79	70.29	58.55	0.66	89.01	78.20	0.87
MDGCN	75.44	79.75	0.69	93.49	95.60	0.93	85.23	75.60	0.83	92.28	88.09	0.91
S-DMM	83.77	90.98	0.79	88.53	94.53	0.87	88.72	84.20	0.87	81.17	82.85	0.78
3DCAE	59.14	71.58	0.51	72.64	75.58	0.70	70.58	60.98	0.66	70.65	81.47	0.68
MDL4OW	76.55	81.42	0.70	82.44	90.57	0.81	87.46	83.41	0.86	93.65	94.98	0.92
GiGCN	93.51	94.07	0.92	97.34	98.34	0.97	92.50	86.52	0.91	97.51	95.88	0.97

Our GiGCN siginificantly outperforms other methods especially when labeled samples are a few.

Note: access code for Baidu Disk is mst1.

1. Create Envirement:

Python 3 (Recommend to use Anaconda)
NVIDIA GPU + CUDA

Python packages:

cd graph-in-graph
pip install -r requirements.txt

2. Data Preparation:

Download the data including raw .mat files and corresponding .jpg files used in superpixel segmentation from here (code: 4zyf) for a quick start and place them in GiGCN/.
Before trainig, every data set is split by runing trainTestSplit.py, shown as follow:
```
python trainTestSplit.py --name PaviaU (data set name)
```

3. Training

To train a model, run

# Training on PaviaU data set
python train.py --name PaviaU --block 100 --gpu 0

Here, --block denots the number of superpixel, which lies in [50, 100, 150, 200] in our ensemble setup.

The model with best accuracy will be saved.

Note: The scikit-image package in our experimental configuaration is of version 0.15.0 whose parameter start_label defaults to 0. However, in the lastest version, it defaults to 1. So when encountering the problem that indexes are out of the bounder at Line 54 in Trainer.py, you should set start_label as 0 explicitly.

4. Prediction:

To test a trained model, run

# Testing on PaviaU data set
python predict.py --name PaviaU --block 100 --gpu 0

The code will load the best model in the last phase automatically.

Citation

If this repo helps you, please consider citing our works:

@ARTICLE{9801664,
  author={Jia, Sen and Jiang, Shuguo and Zhang, Shuyu and Xu, Meng and Jia, Xiuping},
  journal={IEEE Transactions on Neural Networks and Learning Systems}, 
  title={Graph-in-Graph Convolutional Network for Hyperspectral Image Classification}, 
  year={2022},
  volume={},
  number={},
  pages={1-15},
}