Development of a transfer learning-based, multimodal neural network for identifying malignant dermatological lesions from smartphone images
Description
Early skin cancer detection in primary care settings is crucial for prognosis, however primary care physicians often lack relevant training. Machine learning (ML) tools offer a potential solution by enabling systematic triaging of suspicious lesions using images. This study aims to develop a novel neural network for the binary classification of skin lesions into malignant and benign categories using smartphone images and clinical data via a multimodal and transfer learning-based approach. We used the PAD-UFES-20 dataset, which included 2298 sets of images and associated clinical data. Three neural network models were developed: a clinical data network, an image-based network based on a pre-trained DenseNet-121, and a multimodal network combining clinical and image data. Models were tuned using Bayesian Optimization HyperBand (BOHB) across 5-fold cross-validation. The neural network models were built using keras 3.3.3 and tensorflow 2.16.1. BOHB tuning was completed using ray 2.30.0.
Files
Steps to reproduce
The Mendeley Data repository includes model weights generated and used during evaluation (/model_weights), original plots (/plots), and Grad-CAM heatmaps for all internal validation images (/plots/gradcam). The repository also includes the PAD-UPES-20 dataset, available under a Creative Commons 4.0 (CC BY 4.0) license from Pacheco et al. (https://doi.org/10.17632/zr7vgbcyr2.1). The dataset is already split into a derivation set (N=1,298) under /PAD-UFES-20/train and an internal validation set (N=1,000) under /PAD-UFES-20/test. To load the final model weights (/model_weights/final_###_net.weights.h5), first initialize the corresponding Keras model architectures using the build_clinical_network(), build_image_network(), and build_multimodal_network() functions in the lesionutils package. Examples for how to load the final model weights are provided in the explainability Jupyter notebook files included in the repository. It is recommended to run the provided code in a conda virtual environment. Conda can be installed from https://conda.io/projects/conda/en/latest/user-guide/install/index.html. After creating and activating your virtual environment, run `pip install -r requirements.txt` in your system terminal to install the necessary packages. Start by examining the structure of the /PAD-UFES-20 directory. The dataset was split using split_image_set.R. It is not necessary to run this code, but you can examine the code for correctness or adapt it for your own use. Clinical data used for the clinical neural network is saved as clinical_data_train.csv and clinical_data_test.csv, and imputed into clinical_data_train_imputed.csv and clinical_data_test_imputed.csv using the impute_script.R. The dataset is also split based on Fitzpatrick skin phototypes using the split_data_by_fitzpatrick.R script. The Fitzpatrick subsets are available under /fitzpatrick_images. In the project directory, the raytune_clinical.py, raytune_image.py, and raytune_multimodal.py files are used for hyperparameter tuning using BOHB across 5-fold cross-validation. These can be run in your system terminal using `python <script name>`. Before running the tuning scripts, examine variables in lesionutils.constants and the Ray Tune settings in the Python scripts to ensure that caching and temporary directories are set correctly. When the tuning code completes, it will print out the most optimal parameters in the console. Your hyperparameters should be similar to the ones saved in the tuned_trainables() functions in the lesionutils package. Once you have the optimized set of parameters, you can generate the same sets of plots and visualizations that we included in our publications, as well as perform threshold tuning, using the #_explainability, #_results_fitzpatrick, and #_results_visualizations Jupyter notebook files in the project directory. We have kept all of our outputs in the notebook files for your reference.