A Data-set about Surface Damage Identification for Heritage Site Protection: A Mobile Crowdsensing Solution Based on Deep Learning
Description
The dataset has been collected to address the general problem of built heritage protection against both deterioration and loss. In order to continuously monitor and update the structural health status, a crowd-sensing solution based on powerful and automatic deep learning technique is proposed. The aim of this solution is to get rid of the limitations of manual and visual damage detection methods that are costly and time consuming. Instead, automatic visual inspection for damage detection on walls is efficiently and effectively performed using an embedded Convolutional Neural Network (CNN). This CNN detects the most frequent types of surface damage on wall photos. The study has been conducted in the Kasbah of Algiers where the four following types of damage have been considered: Efflorescence, Spall, Crack, and Mold. The CNN is designed and trained to be integrated into a mobile application for a participatory crowd-sensing solution. The application should be widely and freely deployed, so that any user can take a picture of a suspected damaged wall, and get an instant and automatic diagnosis, through the embedded CNN. In this context, we have used MobileNetV2 with a transfer learning approach. A set of real images have been collected and manually annotated, and have been used for training, validation, and test. Extensive experiments have been conducted in order to assess the efficiency and the effectiveness of the proposed solution, using a 5 fold cross validation procedure. Obtained results show in particular a mean weighted average precision of 0.868±0.00862 (with a 99% of confidence level) and a mean weighted average recall of 0.84±0.00729 (with a 99% of confidence level). Obtained results show that our method remains effective even when using a small network and medium to low resolution images. In order to provide the CNN to be deployed, as well as the dataset used for the test, to the researchers, we have made them available online. Also a source code to perform a quick test on Google Collaboratry is available. The model has been saved in Keras "h5" format, and also in a TensorFlow Lite "tflite" format, ready to be deployed on a mobile application. Files published are: - "Test_Kasbah_damage.ipynb": python source code to perform the test, in Google Collaboratry. - "multi_label_CV_final.h5": the CNN saved in a h5 keras format, to be used for the test, and other purposes. - "test_multi_label.zip": the test dataset used in the paper, saved in a zip compressed file (634 images). - "multi_label_CV_final.tflite" : the CNN saved in a tflite format, ready for deployment in a mobile application.
Files
Steps to reproduce
Wall photos have been collected from the Kasbah of Algiers, over six different days, outside houses, after obtaining the required authorization. To the best of our knowledge, this is the first time that such a dataset for surface damage detection has been collected at this specific site. Two mid range phones have been used. The photos have been taken in daylight, but at different times. The images used have been cropped from the original images. The objective is to have in each image only the suspicious surface. As a result, the resolution of the cropped images has been variable. On the other hand, some of the cropped images contain multiple types of damage that are superimposed. These images are very useful in order to assess the ability of the CNN to deal with this multi-label classification problem. This is particularly important as the presence of multiple overlapping damages in a same wall is frequent in the Kasbah of Algiers. A total number of 2674 images have been collected using two smartphone cameras: (1) A single 12 Megapixel camera, with integrated localization information (A-GPS GLONASS, BDS), and (2) a 13 Megapixel single camera. The resolution of resulting cropped images ranges from 224x224 to 1500x1500 pixels. This dataset have been divided into two subsets: A train/validation set of 2040 images, and a test set of 634 images. Only the test dataset of 634 images is published. In this context, labeling consists in associating the identified damages with each image, by an expert. As explained before, several damages can be associated with the same image. Considered damages are the following: Efflorescence, Spall, Crack, Mold. Finally, all cropped images from both train and test sets have been resized to 224x224 pixels.