Experimental Data and Source Code for Oracle-Free Metamorphic Test Case Prioritization for Deep Learning-Based Medical Image Classification
Description
This code and supplementary-material package supports the experiments reported in the manuscript prepared for the Journal of Software: Evolution and Process. The code implements the oracle-free metamorphic test case prioritization workflow for CIFAR-10 and ISIC2019. It includes the proposed UI-SumRank method, which combines uncertainty variation based on Jensen-Shannon divergence and interpretability variation based on Grad-CAM drift, as well as the ablation variants U-only, I-only, and equal-weight UI. The baseline methods DeepGini, QuoTe, and Random are also included. The code supports metamorphic-relation image generation, class-balanced Top-K prioritization, seed-level experiment execution, runtime profiling, and post-hoc statistical analysis. The main scripts include generate_cifar10_isic_mrs.py for generating metamorphic image variants; ui_sumrank_cifar10.py and ui_sumrank_isic2019.py for the proposed method and ablations; cifar10_baselines_all.py and isic2019_baselines_all.py for baseline experiments; batch_run_main.py and batch_run_all_baselines.py for batch execution; runtime_utils.py for runtime logging; and analyze_runtime_and_statistics.py for aggregating runtime, mean, standard deviation, confidence interval, and effect-size results. The supplementary archive JSEP_supplementary_individual_pdfs.zip contains individual PDF files for the supplementary figures and tables associated with the manuscript. It includes Figure S1, which reports Top-50 prioritization result plots; Figure S2, which presents the critical difference diagrams used for non-parametric comparison; Figure S3, which reports the Top-50 ablation plots; and Table S1, which provides the runtime breakdown. These files are provided separately to support journal submission, reviewer inspection, and reproducibility of the reported experimental analysis. Raw CIFAR-10 and ISIC2019 images and trained model checkpoints are not redistributed because they are third-party resources or locally trained artifacts. Users should download or prepare the datasets and checkpoints locally, then use the provided scripts to reproduce the ranking records, seed-level summaries, runtime profiles, aggregated statistical outputs, and supplementary figures/tables.
Files
Steps to reproduce
1. Unzip the code package and enter the extracted folder. 2. Prepare a Python environment with the required packages, including Python 3, PyTorch, torchvision, timm, numpy, scipy, scikit-learn, Pillow, and tqdm. GPU execution is recommended for Grad-CAM-based scoring. 3. Prepare the dataset folders locally. The scripts expect CIFAR-10 and ISIC2019 image folders under a data root, for example: data/cifar10/images/train/<class>/ data/isic2019/Train/<class>/ 4. Generate metamorphic-relation image data: python generate_cifar10_isic_mrs.py --root E:/PythonProject/data --out E:/PythonProject/data_mr --mr_preset lite --max_per_class 0 --seed 42 5. Place the trained checkpoints under the checkpoint folder, for example: E:/PythonProject/ckpts/cifar10_resnet50_9391.pt E:/PythonProject/ckpts/cifar10_convnext_b_9994.pt E:/PythonProject/ckpts/isic2019_resnet50_9094.pt E:/PythonProject/ckpts/isic2019_convnext_b_9421.pt 6. If your local paths are different, edit the path variables in batch_run_main.py and batch_run_all_baselines.py, especially PROJECT_ROOT, DATA_ROOT, MR_ROOT, CKPT_DIR, and OUT_ROOT. 7. Run the proposed UI-SumRank method and its ablations: python batch_run_main.py 8. Run the baseline methods DeepGini, QuoTe, and Random: python batch_run_all_baselines.py 9. After all experiments are complete, aggregate runtime and statistical results: python analyze_runtime_and_statistics.py --main_root E:/PythonProject/out_mjfas --baseline_roots E:/PythonProject/out_mjfas/out_cifar10_baselines E:/PythonProject/out_mjfas/out_isic2019_baselines --out E:/PythonProject/out_mjfas/out_analysis 10. The main outputs include per-seed Top-K ranking files, seed_summary CSV files, runtime_profile JSON/CSV files, runtime summaries, metric summaries, confidence intervals, and effect-size tables. These files can be used to reproduce the experimental tables and runtime/statistical analyses reported in the manuscript.