Franklin Zhao (me:))
Xuefeng Zhao (my undergraduate advisor who supported this project)
Peng Zhao (provided the data-set for this project)
June 2017
-
Developed a damage identification and localization technique for civil structures using convolutional neural networks (based on LeNet-5, AlexNet-8, ZFNet-8, and GoogLeNet(inception v1)-22).
-
Applied region-based convolutional neural networks (Faster-RCNN) to damage detection in real-time.
Note: This project is mostly an application for masonry historic structures using fine-tuned deep learning algorithms. If you are more interested in the CNN architecture or implementation, there might not be much details. This project is aim to provide insights on artificial intelligence for infrastructural engineering.
Manual inspection (i.e., visual inspection and/or with professional equipment) is the most predominant approach for identifying and assessing superficial damage of masonry historic structures at present. However, this method is costly and at times difficult to apply to remote structures or components. Existing convolutional neural network (CNN)‐based damage detection methods have not been specifically designed for the multiple damage identification of masonry historic structures. To overcome these limits, a deep architecture of CNN damage classification techniques for masonry historic structures is proposed in this article using a sliding window‐based CNN method to identify and locate four categories of damage (intact, crack, efflorescence, and spall) with an accuracy of 94.3%. This is the first attempt to identify the multidamage of historic masonry structures based on CNN techniques and achieve excellent classification results. The data are only trained and tested from images of the Forbidden City Wall in China, and the pixel resolutions of stretcher brick images and header brick images are 480 × 105 and 210 × 105, respectively. Two CNNs (AlexNet and GoogLeNet) are both trained on a small dataset (2,000 images for training, 400 images for validation and testing) and a large dataset (20,000 images for training, 4,000 images for validation and testing). The performance of the trained model (94.3% accuracy) is examined on five new images with 1,860 × 1,260 pixel resolutions.
A Few Takeaways: If you are really into the algorithm, I would suggest you take Stanford CS231N first which is a really popular open course. If you are not very confident in Machine Learning, both Stanford CS229 and Berkeley CS189 are good resources. I would also suggest reading those papers presenting deep learning architectures that are among the top ranks in recent CV classification/object detection competitions.
Dataset: You can either use our dataset or create your own dataset! Just take as many pictures of what your are interested in researching as possible. You might also want to use the crowdsourcing technology to let so many other people collect data for you!
Training: The training environment depends on your dataset and CNNs. For example, if your dataset is large (GB/TB/PB), you might consider using distributed workstations instead of PCs, which also applies to deeper CNNs (ResNet, Inception v-4, etc.). You might also need GPU(s) for computing since it is much more faster that CPU-only.
For tools training the dataset, I would suggest using TensorFlow or PyTorch. But if you are only into the application, Caffe is fine. If you choose Caffe, the only thing you need to do is to change the hyperparameters in the .prototxt files.
Application and visualization: If your are going to apply deep learning to real-world engineering problems, only classifing the object in a CV way is not enough. You should figure out how to present or process the output. Also, applicable and reproducible are also very important. For example, you might ask yourself: What is the cost? How is the efficiency? Or does it even work in the real-world problems?
Fig. 1: Schematic diagram of image classification for a brick unit
Fig. 2: Schematic diagram of CNN-based damage identification and classification for a brick unit
Fig. 3: Comparison of fully connected mode and locally connected mode: (a) fully connected mode, and (b) locally connected mode
Fig. 4: Training samples of the 4 categories: (a) intact, (b) spall, (c) crack, and (d) efflorescence for CNN
Fig. 5: Training sample for Faster-RCNN
Fig. 6: Results of training using AlexNet-8: (a) train loss, and (b) validation accuracy
Fig. 7: Results of training using GoogLeNet-22: (a) train loss, and (b) validation accuracy
Fig. 8: Sliding window strategy
Fig. 9: Test results based on GoogLeNet-22 together with sliding window algorithm
Fig. 10: Test results based on Faster-RCNN
For more details see this (Faster-RCNN part is not included).
@article{
title={Damage Classification for Masonry Historic Structures Using Convolutional Neural Networks Based on Still Images},
author={Wang, Niannian and Zhao, Qingan and Li, Shengyuan and Zhao, Xuefeng and Zhao, Peng},
journal={Computer-Aided Civil and Infrastructure Engineering},
publisher={Wiley Online Library}
}