If you don't remember your password, you can reset it by entering your email address and clicking the Reset Password button. You will then receive an email that contains a secure link for resetting your password
If the address matches a valid account an email will be sent to __email__ with instructions for resetting your password
Duke Dermatology, Duke University School of Medicine, Durham, North Carolina, USADepartment of Biostatistics & Bioinformatics, Duke University School of Medicine, Durham, North Carolina, USA
Artificial intelligence (AI) has recently made great advances in image classification and malignancy prediction in the field of dermatology. However, understanding the applicability of AI in clinical dermatology practice remains challenging owing to the variability of models, image data, database characteristics, and variable outcome metrics. This systematic review aims to provide a comprehensive overview of dermatology literature using convolutional neural networks. Furthermore, the review summarizes the current landscape of image datasets, transfer learning approaches, challenges, and limitations within current AI literature and current regulatory pathways for approval of models as clinical decision support tools.
Artificial intelligence (AI) in healthcare is the application of machine learning (ML) algorithms in medical fields to potentially improve diagnosis and predict clinical outcomes (
). The advancements in computing power and vast data curation within health systems have led to algorithm development that can assist healthcare providers as clinical decision-support (CDS) tools. A myriad of AI applications exists within health care such as using electronic health record data for risk predictors (
), and continuous disease monitoring using wearable devices. There have been innovative efforts to procure large numbers of medical image datasets, either within institutions or for public use, such as DeepLesion, which contains 32,000 computed tomography images for scientific studies (
Wang X, Peng Y, Lu L, Lu Z, Bagheri M, Summers RM. ChestX-ray8: hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. arXiv 2017.
).
Computer vision is a field of AI in which the system learns to interpret visual images. It has advanced the process of medical image evaluation with higher accuracy and more efficient analysis (
). The convolutional neural network (CNN) is a type of artificial neural network that has revolutionized image analysis without the need to extract traditional handcrafted features such as colors, intensity value, topological structure, and texture information (
). Researchers have developed deep learning models that have been trained on millions of images for different tasks such as image classification, object detection, and image recognition. Model development for computer vision challenges such as image classification and objection detection is achieved by training and testing on millions of images. These models, most notably inspired by ImageNet (
Deng J, Dong W, Socher R, Li L-J, Li K, Fei-Fei L. Imagenet: a large-scale hierarchical image database. A paper presented at: 2009 IEEE Conference on Computer Vision and Pattern Recognition. 20‒25 June 2009; Miami, FL.
in: Fleet D. Pajdla T. Schiele B. Tuytelaars T. Computer vision – ECCV 2014. ECCV 2014. Lecture notes in computer science. Springer,
Heidelberg, Germany2014: 740-755
Kuznetsova A, Rom H, Alldrin N, Uijlings J, Krasin I, Pont-Tuset J, et al. The open images dataset v4: unified image classification, object detection, and visual relationship detection at scale. arXiv 2020.
Xiao J, Hays J, Ehinger KA, Oliva A, Torralba A. Sun database: large-scale scene recognition from abbey to zoo. A paper presented at: 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. 13‒18 June 2010; San Francisco, CA.
) challenges, can either detect or classify numerous different categories such as dogs or cats in a given image with high accuracy.
Medical imaging field has adapted these CNN methods to solve a diverse array of problems, using datasets obtained from various imaging modalities such as chest x-rays (
). In medical image analysis, the lack of data creates a bottleneck for training a deep learning model. Acquiring and annotating medical images is costly, time consuming, and labor intensive. Data sharing may serve as a potential solution to accelerate data collection, but ethical and privacy issues can hinder institutional data sharing. Hence, transfer learning has vastly improved the medical imaging field by allowing the use of models that have been pretrained on millions of images to solve numerous medical imaging problems, alleviating the need to spend hours building an effective model or collecting vast amounts of clinical data. Pretrained models can be fine tuned to unique problems according to the amount of available data and the data similarity.
Thus, one can choose a standard model, often trained on a popular dataset (such as ImageNet), and fine tune the network to fit a given problem. Contrary to the assumptions that weights from a model pretrained on real-world images may not translate well for medical images, studies have shown that ImageNet-pretrained models have produced human-level accuracy for medical image classification such as in pathology (
High concordance between immunohistochemistry and fluorescence in situ hybridization testing for HER2 status in breast cancer requires a normalized IHC scoring system.
Man against machine: diagnostic performance of a deep learning convolutional neural network for dermoscopic melanoma recognition in comparison to 58 dermatologists.
). In dermatology, AI systems using transfer learning are comparable with or even surpass the performance of dermatologists in diagnosing skin conditions (
Man against machine reloaded: performance of a market-approved convolutional neural network in classifying a broad spectrum of skin lesions in comparison with 96 dermatologists working under less artificial conditions.
). The rapidly growing global burden of skin cancer, rise of teledermatology during the COVID-19 pandemic, and supply‒demand imbalance for dermatologists point to an escalating need to establish effective triaging systems supported by AI for dermatological disease detection and diagnosis. This study aims to provide a comprehensive review of published applications of pretrained models on dermatological images, their associated datasets, their limitations, and their outcomes.
Search strategy
We conducted a query of MEDLINE and PubMed Central databases through PubMed using keywords, including dermatology, deep learning, transfer learning, and convolutional neural network. Studies published from the year 2016 to 2021 were included. We excluded studies on the basis of the following criteria: (i) use of non‒deep learning algorithms, (ii) use of custom algorithms, and (iii) meta-analysis and/or review articles. The search strategies are outlined in Figure 1.
In total, 65 studies were included in the final review. Table 1 summarizes the classification tasks, methodology, and outcome metrics for the CNNs developed for skin conditions. Table 2 provides details of the publicly available dataset investigated in this review. Table 3 classifies the studies according to the type of dataset (i.e., institutional or public) and image (i.e., clinical or dermoscopic).
Table 1Summary of Tasks, Methodology, Dataset, and Performance
Shahin AH, Kamal A, Elattar MA. Deep ensemble learning for skin lesion classification from dermoscopic images. A paper presented at: 2018 9th Cairo International Biomedical Engineering Conference (CIBEC). 20‒22 December 2018; Cairo, Egypt.
Best results AUC of dsc + macro: 0.866 mAP of dsc + macro + meta: 0.729
Lack of patient information/clinical information Adding age, gender, location, and lesion size increased the accuracy Common verification bias in dermatoscopic studies, with only pathologically diagnosed cases included.
A convolutional neural network trained with dermoscopic images performed on par with 145 dermatologists in a clinical melanoma image classification task.
Classification Binary: melanoma versus atypical nevi
ResNet50
Public/clinical Trained with ISIC image archive and HAM1000 dataset Validated with Mclass-Benchmark for clinical images obtained from MED-NODE database
Insufficiency of quality training data Difficulty in fully exploiting the discrimination capability gains of very deep CNNs under the circumstance of limited training data
Harangi, 2017
Segmentation/classification (separate) Multiclass: melanoma, seborrheic keratosis, and nevus
Ensemble of GoogLeNet, AlexNet, ResNet50, and VGG-VD-16
Rezvantalab A, Safigholi H, Karimijeshni S. Dermatologist level dermoscopy skin cancer classification using different deep learning convolutional neural networks algorithms. arXiv 2018.
Classification Binary: melanoma, melanocytic nevi, BCC, benign keratosis, AK and intraepithelial carcinoma, dermatofibroma, vascular lesions, and atypical nevi
Comparing DenseNet 201; ResNet152; Inception, version 3; and InceptionResNet, version 2
Public/dermoscopy HAM10000 Dataset PH2 dataset
AUC of melanoma versus that of basal cell carinoma 94.40% (ResNet152) 99.3% (DensNet 201)
The utilized dataset is highly unbalanced, and also no preprocessing step is applied in this paper. Still the results are promising
Mahbod A, Schaefer G, Wang CL, Ecker R, Ellinger I. Skin lesion classification using hybrid deep neural networks. A paper presented at: ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 12‒17 May 2019; Brighton, United Kingdom.
Classification Binary: malignant versus benign nonpigmented skin lesions
Combined Inception, version 3 (dermoscopic images), and ResNet50 (clinical close-ups) using xgboost
Institutional/dermoscopy and clinical 7,895 dermoscopic and 5,829 close-up images of the training set originated from a consecutive sample of lesions photographed and excised by one author (CR) at a primary skin cancer clinic in Queensland, Australia. Image data split: described in the manuscript in detail
AUC: 0.742 SE: 80.5% SP: 53.5%
Test set included >51 distinct classes, of which most did not have enough examples to be integrated into the training phase
Chang H. Skin cancer reorganization and classification with deep neural network. arXiv 2017.
Segmentation/classification (separate) Binary: malignant versus benign nonpigmented skin lesions
Segmentation: U-Net Classification: Google Inception, version 3
Public/dermoscopy ISIC challenge website. A total of 2,000 dermoscopic images includes 374 melanoma images, 1,372 nevus images, and 254 seborrheic keratosis images
The CNN achieves performance on par with all tested experts across both tasks, showing an AI capable of classifying skin cancer with a level of competence comparable with that of dermatologists
Man against machine: diagnostic performance of a deep learning convolutional neural network for dermoscopic melanoma recognition in comparison to 58 dermatologists.
Classification Binary: melanoma versus melanocytic nevi
Google’s Inception, version 4
Public and Institutional/dermoscopy Trained and validated on ISIC and ISBI 2016 dataset Tested on 300 from the image library of the Department of Dermatology, University of Heidelberg (Heidelberg, Germany) and on 100 images from ISIC and ISBI dataset Image data split: 20% melanoma, 80% benign nevi
300 test set SE: 95% SP: 80% AUC: 0.95 100 Test Set SE: 95% SP: 63.8% AUC: 0.86
Association between surgical skin markings in dermoscopic images and diagnostic performance of a deep learning convolutional neural network for melanoma recognition.
Most images included in this study were derived from fair-skinned patients residing in Germany; therefore, the findings may not be generalized for lesions of patients with other skin types and genetic backgrounds.
Deep-learning-based, computer-aided classifier developed with a small dataset of clinical images surpasses board-certified dermatologists in skin tumour diagnosis.
Vasconcelos CN, Vasconcelos BN. Convolutional neural network committees for melanoma classification with classical and expert knowledge based image transforms data augmentation. arXiv 2017.
Classification Binary: melanoma, seborrheic keratosis, and nevus
Codella NC, Gutman D, Celebi ME, Helba B, Marchetti MA, Dusza SW, et al. Skin lesion analysis toward melanoma detection: a challenge at the 2017 international symposium on biomedical imaging (ISBI), hosted by the international skin imaging collaboration (ISIC). A paper presented at: 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018). 4‒7 April 2018; Washington, DC.
Mishra R, Daescu O. Deep learning for skin lesion segmentation. A paper presented at: 2017 IEEE International Conference on Bioinformatics and Biomedicine (BIBM). 13‒16 November 2017; Kansas City, MO.
Araújo RL, Ricardo de Andrade LR, Rodrigues JJ, Silva RR. Automatic segmentation of melanoma skin cancer using deep learning. Paper presented at: 2020 IEEE International Conference on E-health Networking, Application & Services (HEALTHCOM). 1‒2 March 2021; Shenzhen, China.
Pomponiu V, Nejati H, Cheung NM. Deepmole: deep neural networks for skin mole lesion classification. A paper presented at: 2016 IEEE International Conference on Image Processing (ICIP). 25‒28 September 2016; Phoenix, AZ.
Kaymak S, Esmaili P, Serener A. Deep learning for two-step classification of malignant pigmented skin lesions. A paper presented at: 2018 14th Symposium on Neural Networks and Applications (NEUREL). 20‒21 November 2018; Belgrade, Serbia.
Menegola A, Fornaciali M, Pires R, Bittencourt FV, Avila S, Valle E. Knowledge transfer for melanoma screening with deep learning. A paper presented at: 2017 IEEE 14th International Symposium on Biomedical Imaging (ISBI 2017). 18‒21 April 2017; Melbourne, Australia.
Classification Binary: acral melanoma and benign nevi
Fine-tuned modified VGG model with 16 layers
Institutional/dermoscopy A total of 724 dermoscopy images were collected from January 2013 to March 2014 at the Severance Hospital in the Yonsei University Health System (Seoul, Korea) and from March 2015 to April 2016 at the Dongsan Hospital in the Keimyung University Health System (Daegu, Korea)
Acc: Group A: 83.51% Group B: 80.23% AUC Group A: 0.8 Group B: 0.84
Lopez AR. Giro-i-Nieto X, Burdick J, Marques O. Skin lesion classification from dermoscopic images using deep learning techniques. A paper presented at: 2017 13th IASTED International Conference on Biomedical Engineering (BioMed). 20‒21 February 2017; Innsbruck, Austria.
Augmented intelligence dermatology: deep neural networks empower medical professionals in diagnosing skin cancer and predicting treatment options for 134 skin disorders.
Detection/classification (end-to-end) Binary: Benign versus malignant lesions
Detection: Faster R-CNN Classification: SENet, SE-ResNeXt-50, and SE-ResNet-50
Institutional/clinical 1,106,886 training set, 2,844 validation set, and 325 test set from Asan Medical Center; plastic surgery from Chonnam National University Department of Plastic Surgery and Hallym University Department of Plastic Surgery
AUC: 0.92 SE: 92.5% at t > 0.9 SP: 70.0% at t > 0.9 SE: 80.0% SS at t > 0.8 SP: 87.5% at t > 0.8
Algorithm was validated with one race (Asian) within one region (South Korea). Model only has photo information and lacks other evaluations from physicians.
Segmentation Automated prediction of lesion segmentation boundaries from dermoscopic images
ResNet-Inception, version 2; DeepLab, version 3 Ensemble-A: combines results from both models Ensemble-L: chooses the larger area Ensemble-S: chooses a smaller area
Classification Design and evaluation of a smart psoriasis identification system based on clinical images
DenseNet; Inception, version 3; InceptionResNet, version 2; and Xception Inception, version 3, performed best
Institutional/clinical Images collected by dermatologists at Xiangya Hospital, Annotated by three dermatologists with >10 years’ experience at Xiangya Hospital according to the corresponding medical record and pathology results
AUC: 0.981 ± 0.015 SE: 0.98 SP: 0.92
Model has the capability to identify psoriasis (acc of model: 0.96) with a level of competence comparable with those of 25 dermatologists (mean acc of 25 dermatologists: 0.87)
Segmentation CNN model for detection of psoriasis lesions
U-Net
Institutional/dermoscopy Images captured and annotated by a dermatologist at the Psoriasis Clinic and Research Centre, Psoriatreat, Pune, Maharashtra, India
Segmentation Automatic approach based on a deep learning model using transfer learning for the segmentation of psoriasis lesions from the digital images of different body regions of patients with psoriasis.
U-Net
Institutional/clinical Psoriasis Clinic and Research Centre, Psoriatreat, Pune, Maharashtra, India
Classification Train an efficient deep-learning network to recognize dermoscopic images of psoriasis (and other papulosquamous diseases), improving the Acc of the diagnosis of psoriasis
EfficientNet-b4
Institutional/dermoscopy 7,033 dermoscopic images from 1,166 patients collected from the Department of Dermatology, Peking Union Medical College Hospital (Peking, China)
The algorithm only recognized the dermoscopic images from lesions to diagnose the disease, different from the clinical diagnosis process with multimodal data (e.g., age, sex, medical history, and treatment response) and more types of diseases involved. The model may not perform well on other populations with different skin types/colors.
Institutional/clinical 203 photographs of Caucasian patients aged between 18 and 80 years and suffering from plaque-type psoriasis were selected. The photographs included were taken with a Nikon D700 camera
Acc: 0.91 F1-score: 0.71
Restriction of the data is the inclusion of mostly Caucasian patients. Because the manifestation of psoriasis differs depending on the skin type, including only a few images of other skin types would have led to a highly imbalanced data set
Classification Automatically classify psoriasis-affected skin area from normal healthy skin using machine learning algorithm
mobilenet, nasnetlarge NasNetLarge chosen
Institutional/clinical Psoriasis: Department of Skin and STD, Karnataka Institute of Medical Sciences (Hubli, India) and Department of Dermatology, Navodaya Medical College (Raichur, India) Normal: Department of Computer Science, Rani Channamma University (Belagavi, India).
Deep neural networks show an equivalent and often superior performance to dermatologists in onychomycosis diagnosis: automatic construction of onychomycosis datasets by region-based convolutional deep neural network.
Classification Binary: onychomycosis versus nononychomycosis
Used CNN (ResNet152) to select hand and foot images R-CNN (VGG16) to select nail parts Ensemble method using two CNN (ResNet152 + VGG19) for feature extraction and feedforward neural network for classification
Institutional/dermoscopy Clinical images obtained from four hospitals Trained with Asan dataset Validated with a dataset from Inje University, Hallym University, and Seoul National University
SE/SP/AUC B1 Dataset 96.0/94.7/0.98 B2 Dataset 82.7/96.7/0.95 C Dataset 82.3/79.3/0.93 D Dataset 87.7/69.3/0.82
The clinical photographs used in dermatology are not standardized in terms of image composition. Additional medical photographs may be required for accurate medical diagnoses, and retrieving sufficient numbers of such images may be difficult or even impossible in practical terms
Goyal M, Yap MH, Reeves ND, Rajbhandari S, Spragg J. Fully convolutional networks for diabetic foot ulcer segmentation. A paper presented at: 2017 IEEE International Conference on Systems, Man, and Cybernetics (SMC). 5‒8 October 2017; Banff, Alabama, Canada.
Yap MH, Cassidy B, Pappachan JM, O’Shea C, Gillespie D, Reeves ND. Analysis towards classification of infection and ischaemia of diabetic foot ulcers. A paper presented at: 2021 IEEE EMBS International Conference on Biomedical and Health Informatics (BHI). 27‒30 July 2021; Athens, Greece.
Brüngel R, Friedrich CM. DETR and YOLOv5: exploring performance and self-training for diabetic foot ulcer detection. A paper presented at: 2021 IEEE 34th International Symposium on Computer-Based Medical Systems (CBMS). 7‒9 June 2021; Aveiro, Portugal.
Classification Rosacea, acne, seborrheic dermatitis, and eczema
ResNet50
Institutional/clinical 24,736 photos comprising 18,647 photos of patients with rosacea and 6,089 photos of patients with other skin diseases such as acne, facial seborrheic dermatitis, and eczema
Rosacea Detection Acc: 0.914 Precision: 0.898 AUC: 0.972 Rosacea versus Acne Acc: 0.931 Precision: 0.893 Rosacea versus seborrheic dermatitis and eczema Acc: 0.757 Precision: 0.667
One single dermoscopic image covers only a small proportion of the whole lesion, which hardly represents all the clinical characteristics of the disease comprehensively. Integrating different types of images (clinical, dermoscopic, histopathological) could improve performance.
Symptoms such as itching, pain, and other clinical symptoms are absent in the image analysis, which can help the dermatologist in diagnosing the disease
Classification Multiclass: 26 disease classes (common skin conditions, representing roughly 80% of the volume of skin conditions seen in a primary care setting)
Inception, version 4
Institutional/dermoscopy 16,114 deidentified cases (photographs and clinical data) from a teledermatology practice serving 17 sites
Validation Set A: Top 1: SE: 0.48, PPV: 0.96 Top-3: SE: 0.88, PPV: 0.69 Validation set B: Top 1: SE: 0.57, PPV: 0.95 Top 3: SE: 0.92, PPV: 0.76
Dataset was deidentified, and only structured meta-data was available, which loses information compared with free text clinical notes or an in-person examination
Man against machine reloaded: performance of a market-approved convolutional neural network in classifying a broad spectrum of skin lesions in comparison with 96 dermatologists working under less artificial conditions.
Classification Multiclass: 10 disease classes (nevus, angioma/angiokeratoma, SK, dermatofibroma, solar lentigo, AK, Bowen’s disease, melanoma, BCC, and SCC)
CNN architecture based on inception, version 4
Public/dermoscopy Tested on MSK-1 dataset (1,100 images) and the ISIC 2018 challenge 16 dataset (1,511 images)
SE: 95.0% SP: 76.7% AUC: 0.918
The test set did not include some other benign (e.g., viral warts), malignant (e.g., Merkel cell carcinoma), or inflammatory (e.g., clear cell acanthoma) skin lesions. Therefore, our results should not be generalized to a large prospective patient population. Dermoscopic images were mostly of patients with a Caucasian genetic background and may not provide comparable results in a population of nonwhite skin types.
A benchmark for automatic visual classification of clinical skin disease images.
in: Leibe B. Matas J. Sebe N. Welling M. Computer vision - ECCV 2016. ECCV 2016. Lecture Notes in Computer Science. Springer,
Heidelberg, Germany2016: 206-222
Public/dermoscopy and Clinical SD-198, which contains 198 different diseases from different types of eczema, acne, and various cancerous conditions. There are 6,584 images in total
CaffeNet: 46.69% VGGNet: 50.27%
The dataset shows an imbalance among different categories. Authors tried to collect the same number of samples, whereas some diseases rarely appear in real life
Classification Acne, rosacea, psoriasis, eczema, and cutaneous
VGG 16
Institutional/clinical A total of 19,641 images were provided from the local skin image database of the Department of Dermatology, Aarhus University Hospital (Aarhus, Denmark)
Acne versus Rosacea: SE: 85.42% SP: 89.53% Cutaneous versus Eczema: SE: 74.29% SP: 84.09% Psoriasis versus Eczema: SE: 81.79% SP: 73.57%
One limitation is racial bias, as the data source consisted primarily of Patients with Fitzpatrick skin type II and III. Concerns have been raised about racial bias in CAD in dermatology because databases used for machine learning have historically had an overrepresentation of Caucasian data
Classification Binary: malignant versus benign lip disorders
Inception-ResNet, version 2
Institutional/dermoscopy Image label split: a total of 1,629 SNUH images (743 malignant and 886 benign) for the training set. The remaining 344 SNUH images (110 malignant and 234 benign) were used as the testing set, along with 281 images (57 malignant and 224 benign) from Seoul National University Bundang Hospital (225 images) and SMG-SNU Boramae Medical Center
Limitations: The algorithm was used to classify binary responses of the diseases and not the likelihood rating from the participants. Most of the images used in this study were of Asian people. The diversity of the diagnoses from the external data of the two affiliated hospitals was lower than that of the training set. The dataset was small compared with those used in previous studies. It is difficult to obtain high-quality, diagnosis-annotated lip images, but these obstacles can be overcome if more appropriate images become available in the future.
Multi-resolution-tract CNN with hybrid pretrained and skin-lesion trained layers.
in: Wan L. Adeli E. Wang Q. Shi Y. Suk H.I. Machine learning in medical imaging. MLMI 2016. Lecture Notes in Computer Science. Springer,
Heidelberg, Germany2016: 164-171
A hybrid of the pretrained AlexNet architecture for early network layers and additional untrained layers for later network layers that learn only from skin images.
Abbreviations: Acc., accuracy; AI, artificial intelligence; AK, actinic keratosis; AUC, area under the curve; BCC, basal cell carcinoma; CAD, computer aided diagnostic; CNN, convolutional neural network; Coeff, coefficient; DF, dermatofibroma; DFU, diabetic foot ulcer; DI, dice coefficient; dsc, dermatoscopic; IEC, intraepithelial carcinoma; ISBI, international symposium on biomedical imaging; ISIC, International Skin Imaging Collaboration; JA, jaccard index; mAP, mean average precision; MCC, Matthews correlation coefficient; N/A, not applicable; NPV, negative predictive value; PPV, positive predictive value; PYO, pyogenic granuloma; SCC, squamous cell carcinoma; SE, standard error; SK, seborrhoeic keratosis; SMG-SNU, Seoul Metropolitan Goverment-Seoul National University; SNUH, Seoul National University Hospital; SP, specificity; SVM, support vector machine; TTA, test-time augmentation.
7 Bi L, Kim J, Ahn E, Feng D. Automatic skin lesion analysis using large-scale dermoscopy images and deep residual networks. arXiv 2017.
8 Li KM, Li EC. Skin lesion analysis towards melanoma detection via end-to-end deep learning of convolutional neural networks. arXiv 2018.
9 Rezvantalab A, Safigholi H, Karimijeshni S. Dermatologist level dermoscopy skin cancer classification using different deep learning convolutional neural networks algorithms. arXiv 2018.
10 Chang H. Skin cancer reorganization and classification with deep neural network. arXiv 2017.
11 Mirunalini P, Chandrabose A, Gokul V, Jaisakthi S. Deep learning for skin lesion classification. arXiv 2017.
12 Murphree DH, Ngufor C. Transfer learning for melanoma detection: participation in ISIC 2017 skin lesion classification challenge. arXiv 2017.
13 Vasconcelos CN, Vasconcelos BN. Convolutional neural network committees for melanoma classification with classical and expert knowledge based image transforms data augmentation. arXiv 2017.
14 Sousa RT, de Moraes LV. Araguaia medical vision lab at ISIC 2017 skin lesion classification challenge. arXiv 2017.
15 Yang X, Zeng Z, Yeo SY, Tan C, Tey HL, Su Y. A novel multi-task deep learning model for skin lesion segmentation and classification. arXiv 2017.
16 Goyal M, Hassanpour S. A refined deep learning architecture for diabetic foot ulcers detection. arXiv 2020.
17 Galdran A, Carneiro G, Ballester MAG. Convolutional nets versus vision transformers for diabetic foot ulcer classification. arXiv 2022.
Task 1: Lesion Segmentation Training Data: 900 dermoscopic lesion images in JPEG format, with EXIF data stripped Training Ground Truth: 900 binary mask images in PNG format Test Data: 379 images of the same format as the training data Task 2: Detection and Localization of Visual Dermoscopic Features/Patterns Training Data: 807 lesion images in JPEG format and 807 corresponding superpixel masks in PNG format Training Ground Truth: 807 dermoscopic feature files in JSON format Test Data: 335 images of the exact same format as the training data Task 3: Disease Classification Training Data: 900 lesion images in JPEG format Training Ground Truth: 900 entries of gold standard malignant status Test Data: 379 images of the exact same format as the training data Image Label Split: Task 3 (727 benign, 173 malignant)
For all three tasks: Training Dataset: 2,000 lesion images in JPEG format and 2,000 corresponding superpixel masks in PNG format, with EXIF data stripped Training ground truth: 2,000 binary mask images in PNG format, 2,000 dermoscopic feature files in JSON format, 2,000 entries of gold standard lesion diagnosis Validation Dataset: 150 images Test dataset: 600 images Image Label Split: Melanoma 374, seborrheic keratosis 254, other (benign): 1,372
For Tasks 1 and 2: Training Dataset: 2,594 images and 12,970 corresponding ground truth response masks (five for each image). Validation Dataset: 100 images Test dataset: 1,000 images For Task 3 (HAM10000 Dataset): Training Dataset: 10,015 images and 1 ground truth response CSV file (containing one header row and 10,015 corresponding response rows). 10,015 entries grouping each lesion by image and diagnosis confirm the type. Training ground truth: 2,000 binary mask images in PNG format, 2,000 dermoscopic feature files in JSON format, and 2,000 entries of gold standard lesion diagnosis Validation Dataset: 193 images Test dataset: 1,512 images Image Label Split: Actinic keratoses and intraepithelial carcinoma/Bowen's disease (327 images), basal cell carcinoma (514 images), benign keratosis-like lesions (1,099 images), dermatofibroma (115 images), melanoma (1,113 images), melanocytic nevi (6,705 images), and vascular lesions (142 images)
Training Set: 25,331 JPEG images of skin lesions and metadata entries of age, sex, and general anatomic site with gold standard lesion diagnosis Test Set: 8,238 JPEG images of skin lesions and metadata entries of age, sex, and general anatomic site. Image Label Split: Actinic keratoses and intraepithelial carcinoma/Bowen's disease (867 images), basal cell carcinoma (3,323 images), benign keratosis‒like lesions (2,624 images), dermatofibroma (239 images), melanoma (4,522 images), melanocytic nevi (12,875 images), and vascular lesions (253 images)
Training Set: 33,126 DICOM images with embedded metadata and metadata entries of patient ID, sex, age, and general anatomic site with gold standard lesion diagnoses. Test Set: 10,982 DICOM images with embedded metadata and metadata entries of patient ID, sex, age, and general anatomic site. Image Label Split: Benign keratosis-like lesions (37 images), Lentigo (44 images), solar lentigo (7 images), melanoma (584 images), melanocytic nevi (5193 images), seborrheic keratoses (135 images) and other/unknown (benign) (27,124 images)
The dataset includes over 2,000 clinical and dermoscopy color images, along with corresponding structured metadata Image Label Split: basal cell carcinoma (42 images), nevi (575 images), dermatofibroma (20 images), lentigo (24 images), melanoma (268 images), miscellaneous (8 images), seborrheic keratoses (45 images), and vascular lesion (29 images) Only 1,011 labels are shown in the dataset
4,000 images, with 2,000 used for the training set and 2,000 used for the testing set. An additional 200 images were used for sanity checking. The training set consists of DFU images only, and the testing set comprised images of DFU and other foot/skin conditions and images of healthy feet. The dataset is heterogeneous, with aspects such as distance, angle, orientation, lighting, focus, and the presence of background objects all varying between photographs. Image Label Split: Not reported unless requested
15,683 DFU patches, with 5,955 training, 5,734 for testing, and 3,994 unlabeled DFU patches Image Label Split: Training set (2,555 infections only, 227 ischemia only, 621 both infection and ischemia, and 2,552 without ischemia and infection)
We found 22 different types of the pretrained models used for dermatology application, with ResNet being the most widely used, followed by Inception and VGG. A total of 45 studies conducted classification tasks across different skin diseases, including melanoma, foot ulcer, psoriasis, and rosacea. A total of nine studies focused on skin lesion segmentation and two studies focused on skin lesion detection (bounding box) mostly using U-Net architecture, which is well-suited for object detection tasks. A total of six studies tackled both segmentation and classification tasks separately, whereas three studies aimed to develop an end-to-end model from segmentation to classification.
Discussion
Model selection and feature extraction approaches
The classification methods can be divided into two approaches: single deep learning models and ensemble methods (
Single deep learning models, as the name implies, use a single pretrained model without modification of the architecture. Often, the studies tested multiple models and report on the one with the best performance. For instance,
investigated five different models, such as VGG16, ResNet-101, InceptionV3, DenseNet121, and EfficientNet, for classifying infection and ischemia of diabetic foot ulcers and reported that the EfficientNetB0 had the best results.
evaluated various submodels of VGG, ResNet, EfficientNet, DenseNet, Inception, and MobileNet for binary melanoma classification and discovered that EfficientNetB7 had the highest accuracy of 99.33%.
The ensemble method combines predictions from two or more models that could improve the predictive performance instead of a single model (
Harangi B. Skin lesion detection based on an ensemble of deep convolutional neural network. arXiv 2017.
used an ensemble of GoogLeNet, AlexNet, ResNet-50, and VGG-VG-16 for melanoma classification and showed that the ensemble method outperforms each individual deep learning method.
Deep neural networks show an equivalent and often superior performance to dermatologists in onychomycosis diagnosis: automatic construction of onychomycosis datasets by region-based convolutional deep neural network.
used an ensemble of ResNet-152 and VGG19 for onychomycosis diagnosis and achieved the highest classification performance than the dermatologists.
Few studies used pretrained models to extract features and apply other traditional classification algorithms such as support vector machine (SVM) or XGBoost.
Mahbod A, Schaefer G, Wang CL, Ecker R, Ellinger I. Skin lesion classification using hybrid deep neural networks. A paper presented at: ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 12‒17 May 2019; Brighton, United Kingdom.
used AlexNet, VGG16, and ResNet-18 as feature extractors and used an SVM as a classifier for each network. Each SVM score is fused to obtain a probability for binary classification.
used ResNet-50 to extract features and averaged the scores from a neural network classification layer and the SVM classifier to obtain the final prediction.
combined outputs from InceptionV3 and ResNet-50 and used XGBoost to compute the probabilities.
Datasets in dermatology
The key to developing a high-performance deep learning model is the data. If the number of high-quality datasets is large, there is a higher likelihood that the models will learn to generate more accurate predictions. Several publicly available skin image datasets are provided to engage both dermatology and ML communities to develop novel or to hone existing algorithms. For example, the International Skin Imaging Collaboration (ISIC) archive is one of the most well-known public skin cancer image datasets that has gained a high reputation over the years through algorithmic challenges such as lesion segmentation, visual dermoscopic feature detection and localization, and disease classification since 2016 (
Codella NC, Gutman D, Celebi ME, Helba B, Marchetti MA, Dusza SW, et al. Skin lesion analysis toward melanoma detection: a challenge at the 2017 international symposium on biomedical imaging (ISBI), hosted by the international skin imaging collaboration (ISIC). A paper presented at: 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018). 4‒7 April 2018; Washington, DC.
Codella NCF, Nguyen Q-B, Pankanti S, Gutman DA, Helba B, Halpern AC, et al. Deep learning ensembles for melanoma recognition in dermoscopy images. arXiv 2017.
). The archive contains over 13,000 dermoscopic images collected from leading clinical centers internationally. Additional image repositories include dermatology atlases that were originally used for educational purposes but have recently been used as a database of digital images for algorithm development (Tables 2 and 3). There are few datasets that are available upon a fee and a licensing agreement such as Dermofit Image Library or an ethical committee/institutional approval (
). Other public datasets of different skin diseases include the diabetic foot ulcer challenge, providing >15,000 images of diabetic foot ulcers, other foot/skin conditions, and healthy feet taken with three digital cameras. Besides these public datasets, many clinical institutions have collected their own respective datasets for diseases such as psoriasis, rosacea, and lip disorder.
noted that several manuscripts using the ISIC dataset had duplicate or similar images within training and test sets, introducing bias into the CNN model, and proposed a methodology to remove duplicate images. Because the model predictions improve in accuracy by extracting a higher number of unique features rather than by simply enriching the data by sourcing a large number of images from a small number of sources, it must be noted that a rich source of nonduplicated data must be used for training CNNs to avoid bias and overestimation of model performance.
Data quality and imbalance
The quality of the images can be a cause for concern, especially with nonpublic institutional datasets, because clinical image quality can vary depending on the device and the operator capturing the images. For dermoscopic images, the images are taken with a designated device, and thus the quality may not vary as much. Quality control of images in large datasets is a challenging task. This issue is compounded by overall lack of large image repositories in dermatology; hence, we note that the largest body of literature is in the melanoma binary classification and diabetic foot ulcer models, given the availability of standardized and publicly available datasets. Increasing the diversity of images in datasets and the development of tools to assess image quality and remove duplicate data are solutions needed to improve model development in the future.
Generalizability of models
Although these studies show a potential use of AI models in dermatology, it should be noted that the majority of papers are largely proof of concept, trained, and tested on retrospective datasets. The limitation in generalizability can be broken down into three categories: lack of datasets in general, lack of diversity in datasets, and lack of patient information. The barriers to generalizability would be the data imbalance across age, sex, ethnicity, skin tone, disease type, and disease prevalence, which if not sufficiently addressed could lead to poor performance of the models when tested outside of their training and test population. For reference, image label splits are noted in Table 2 for standardized, publicly available datasets for comparison with disease prevalence in real-world clinical settings.
One study reported that several ML algorithms may underperform on images from patients with skin of color because the datasets used to train these models such as the ISIC challenge archive have been collected heavily from fair-skinned patients in the United States, Europe, and Australia (
). A case study in Uganda showed that only 17% of the images from Fitzpatrick 6 skin (black‒dark) type were correctly diagnosed dermatological conditions through First Derm’s Skin Image Search algorithm, indicating that the model was mainly trained on Caucasian skin types (Kamulegeya et al., 2019
Kamulegeya LH, Okello M, Bwanika JM, Musinguzi D, Lubega W, Rusoke D, et al. Using artificial intelligence on dermatology conditions in Uganda: a case for diversity in training data sets for machine learning. bioRxiv 2019.
Association between surgical skin markings in dermoscopic images and diagnostic performance of a deep learning convolutional neural network for melanoma recognition.
Man against machine reloaded: performance of a market-approved convolutional neural network in classifying a broad spectrum of skin lesions in comparison with 96 dermatologists working under less artificial conditions.
stated that their dataset did not include some other benign, malignant, or inflammatory skin lesions and that the dataset consisted of images from the Caucasian genetic background. Together, these results show that the models will likely not generalize across nonwhite skin types and populations with skin lesion types not included in the dataset used to construct the tested models. To solve these challenges, studies deploying models for prospective validation in real-world settings in which the models will be used are needed. Rigorous validation of models in real-world settings, with training and test data mirroring pretest probability of disease conditions and demographics, will help in generalizability. It should be noted that in current literature, there is a lack of calibration metrics for these AI models. If disease prevalence in the population is known, the model threshold may need to be altered to create a most favorable outcome of data that can meaningfully inform the clinical decision process.
Importance of true labels and ground truth
In larger datasets (MNIST, CIFAR, and ImageNet), deep learning models are able to generalize from training data when true labels far outnumber the incorrect labels (Rolnick et al., 2017
Rolnick D, Veit A, Belongie S, Shavit N. Deep learning is robust to massive label noise. arXiv 2017.
). However, in medical datasets, because of the typical smaller sample sizes, it is unclear whether this holds true. For example, neural network training for true melanoma detection from pigmented lesion biopsies by dermatologists is only 9.60 (95% confidence interval = 6.97‒13.41) by meta-analysis (
). This highlights the importance of using histopathological reports as ground truth for important tasks such as melanoma detection, given that incorrect labels by experts can dilute the dataset, hence creating an inferior model performance. Quantification tasks pose unique challenges such as determining ground truth when there is inter-rater variability from multiple experts, especially in quantification tasks such as inflammatory (urticaria, eczema, etc.) or pigmentation (melasma, post-inflammatory hyperpigmentation) tasks.
Role of clinical information
Most current models are only trained with skin images without consideration of other clinical information related to the patients. Given that physicians usually make clinical decisions with additional information other than imaging, such as with chart reviews, adding this information to the deep learning model could lead to better classification performance.
Man against machine reloaded: performance of a market-approved convolutional neural network in classifying a broad spectrum of skin lesions in comparison with 96 dermatologists working under less artificial conditions.
showed that with the addition of the clinical information, there is an increase in the sensitivity of dermatologists’ management decisions (89.0‒94.1%) and the sensitivity and specificity of diagnostic performance (sensitivity of 83.8‒90.6%, specificity of 77.6‒82.4%). Thus, it is predicted that deep learning models can benefit from the inclusion of patient metadata.
AI versus human performance
Several studies show comparable diagnostic classification results of AI with those of human experts. However, several algorithms suffer from poor generalizability because of the variable performance of the models when tested outside its experimental conditions (
). This leads to cases of faulty AI, which can have a detrimental impact on the trust and promise that researchers and clinicians have for AI in the realm of dermatology.
Although AI-based classification systems cannot replace human experts, they can cooperate with experts and empower them to make accurate skin diagnoses (
Augmented intelligence dermatology: deep neural networks empower medical professionals in diagnosing skin cancer and predicting treatment options for 134 skin disorders.
reported that their model was able to improve the top one accuracy of four dermatologists by 7% in multiclass classification of 134 skin conditions and increase the sensitivity and specificity of malignancy prediction of 47 dermatologists/dermatology residents by 12 and 1%, respectively.
trained a CNN model using over 11,000 dermoscopic images to perform multiclass classification of five skin conditions and found that the mean combined AI‒human accuracy was 83%, which was 1.4% higher than AI alone and 40% higher than experts alone.
AI cannot only assist dermatologists directly but could also be helpful in triage and referral workflow. One study developed a risk-aware neural network model augmented with Bayesian deep networks that showed a 90% prediction accuracy in the diagnosis prediction of seven types of skin lesions and made referrals to experts for only 35% of the tested cases (
Development and assessment of an artificial intelligence–based tool for skin condition diagnosis by primary care physicians and nurse practitioners in teledermatology practices.
). A total of 40 board-certified clinicians were tasked with diagnosing over 1,000 cases with and without AI assistance, and their results were compared with the reference diagnoses made by dermatologists. The study showed that diagnostic agreement for the primary care physicians increased by 10% and that for nurse practitioners increased by 12% with AI assistance.
Regulatory pathway for approval
The Food and Drug Administration (FDA) is the regulatory entity for approval of any models, typically using the Software as a Medical Device (SaMD) 501K regulatory pathway. FDA has different marketing pathways further explored at https://www.fda.gov/medical-devices/device-advice-comprehensive-regulatory-assistance/how-study-and-market-your-device. Most models are marketed as class II (moderate risk), including those that are CDS tools. Devices that make a definitive diagnosis are often class III (highest risk). The more impact the software has on the healthcare diagnosis/treatment decision, the higher the class attributed to it, a concept that is further explored in the IMDRF software as a medical device risk framework (
). For approval, the FDA is currently piloting a program for precertification. The proposed concept entails that an FDA review will change on the basis of the risk level of the device; whether it is an initial product review, major change, or minor change; and depending on the organizational experience. All organizations would have to undergo an organizational excellence review to use this program (
Traditionally, after FDA approval and before marketing, the SaMD must be locked, prohibitive to the adaptive nature of AI/ML software. A discussion paper proposing a novel framework outlining when to submit SaMD modifications can be found (
U.S. Food & Drug Administration Proposed regulatory framework for modifications to artificial intelligence/machine learning (AI/ML)-based software as a medical device (SaMD).
) may be useful to review a current list of approved AI/ML devices.
Conclusion and Future Directions
Deep learning has immense potential in dermatology as an assistive diagnostic tool for skin diseases, with promising value in assisting diagnostic and disease quantification tasks. Clinical use spans clinical care, teledermatology, triaging care, and clinical trials among others. The most pressing challenge preventing AI from being more widely used in dermatology is the lack of diversity in datasets and generalizability studies. Working together with physicians and healthcare providers, these AI algorithms can provide more accurate diagnoses and better care, reduce labor costs and workload, and benefit the healthcare industry overall.
Araújo RL, Ricardo de Andrade LR, Rodrigues JJ, Silva RR. Automatic segmentation of melanoma skin cancer using deep learning. Paper presented at: 2020 IEEE International Conference on E-health Networking, Application & Services (HEALTHCOM). 1‒2 March 2021; Shenzhen, China.
A convolutional neural network trained with dermoscopic images performed on par with 145 dermatologists in a clinical melanoma image classification task.
Brüngel R, Friedrich CM. DETR and YOLOv5: exploring performance and self-training for diabetic foot ulcer detection. A paper presented at: 2021 IEEE 34th International Symposium on Computer-Based Medical Systems (CBMS). 7‒9 June 2021; Aveiro, Portugal.
Codella NC, Gutman D, Celebi ME, Helba B, Marchetti MA, Dusza SW, et al. Skin lesion analysis toward melanoma detection: a challenge at the 2017 international symposium on biomedical imaging (ISBI), hosted by the international skin imaging collaboration (ISIC). A paper presented at: 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018). 4‒7 April 2018; Washington, DC.
Deng J, Dong W, Socher R, Li L-J, Li K, Fei-Fei L. Imagenet: a large-scale hierarchical image database. A paper presented at: 2009 IEEE Conference on Computer Vision and Pattern Recognition. 20‒25 June 2009; Miami, FL.
Deep-learning-based, computer-aided classifier developed with a small dataset of clinical images surpasses board-certified dermatologists in skin tumour diagnosis.
High concordance between immunohistochemistry and fluorescence in situ hybridization testing for HER2 status in breast cancer requires a normalized IHC scoring system.
Goyal M, Yap MH, Reeves ND, Rajbhandari S, Spragg J. Fully convolutional networks for diabetic foot ulcer segmentation. A paper presented at: 2017 IEEE International Conference on Systems, Man, and Cybernetics (SMC). 5‒8 October 2017; Banff, Alabama, Canada.
Man against machine: diagnostic performance of a deep learning convolutional neural network for dermoscopic melanoma recognition in comparison to 58 dermatologists.
Man against machine reloaded: performance of a market-approved convolutional neural network in classifying a broad spectrum of skin lesions in comparison with 96 dermatologists working under less artificial conditions.
Deep neural networks show an equivalent and often superior performance to dermatologists in onychomycosis diagnosis: automatic construction of onychomycosis datasets by region-based convolutional deep neural network.
Augmented intelligence dermatology: deep neural networks empower medical professionals in diagnosing skin cancer and predicting treatment options for 134 skin disorders.
Development and assessment of an artificial intelligence–based tool for skin condition diagnosis by primary care physicians and nurse practitioners in teledermatology practices.
Multi-resolution-tract CNN with hybrid pretrained and skin-lesion trained layers.
in: Wan L. Adeli E. Wang Q. Shi Y. Suk H.I. Machine learning in medical imaging. MLMI 2016. Lecture Notes in Computer Science. Springer,
Heidelberg, Germany2016: 164-171
Kaymak S, Esmaili P, Serener A. Deep learning for two-step classification of malignant pigmented skin lesions. A paper presented at: 2018 14th Symposium on Neural Networks and Applications (NEUREL). 20‒21 November 2018; Belgrade, Serbia.
in: Fleet D. Pajdla T. Schiele B. Tuytelaars T. Computer vision – ECCV 2014. ECCV 2014. Lecture notes in computer science. Springer,
Heidelberg, Germany2014: 740-755
Lopez AR. Giro-i-Nieto X, Burdick J, Marques O. Skin lesion classification from dermoscopic images using deep learning techniques. A paper presented at: 2017 13th IASTED International Conference on Biomedical Engineering (BioMed). 20‒21 February 2017; Innsbruck, Austria.
Mahbod A, Schaefer G, Wang CL, Ecker R, Ellinger I. Skin lesion classification using hybrid deep neural networks. A paper presented at: ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 12‒17 May 2019; Brighton, United Kingdom.
Menegola A, Fornaciali M, Pires R, Bittencourt FV, Avila S, Valle E. Knowledge transfer for melanoma screening with deep learning. A paper presented at: 2017 IEEE 14th International Symposium on Biomedical Imaging (ISBI 2017). 18‒21 April 2017; Melbourne, Australia.
Mishra R, Daescu O. Deep learning for skin lesion segmentation. A paper presented at: 2017 IEEE International Conference on Bioinformatics and Biomedicine (BIBM). 13‒16 November 2017; Kansas City, MO.
Pomponiu V, Nejati H, Cheung NM. Deepmole: deep neural networks for skin mole lesion classification. A paper presented at: 2016 IEEE International Conference on Image Processing (ICIP). 25‒28 September 2016; Phoenix, AZ.
Shahin AH, Kamal A, Elattar MA. Deep ensemble learning for skin lesion classification from dermoscopic images. A paper presented at: 2018 9th Cairo International Biomedical Engineering Conference (CIBEC). 20‒22 December 2018; Cairo, Egypt.
A benchmark for automatic visual classification of clinical skin disease images.
in: Leibe B. Matas J. Sebe N. Welling M. Computer vision - ECCV 2016. ECCV 2016. Lecture Notes in Computer Science. Springer,
Heidelberg, Germany2016: 206-222
Association between surgical skin markings in dermoscopic images and diagnostic performance of a deep learning convolutional neural network for melanoma recognition.
Xiao J, Hays J, Ehinger KA, Oliva A, Torralba A. Sun database: large-scale scene recognition from abbey to zoo. A paper presented at: 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. 13‒18 June 2010; San Francisco, CA.
Yap MH, Cassidy B, Pappachan JM, O’Shea C, Gillespie D, Reeves ND. Analysis towards classification of infection and ischaemia of diabetic foot ulcers. A paper presented at: 2021 IEEE EMBS International Conference on Biomedical and Health Informatics (BHI). 27‒30 July 2021; Athens, Greece.
00058-3&id=gr1.jpg){kind=link}