Advertisement

Development and narrow validation of computer vision approach to facilitate assessment of change in pigmented cutaneous lesions

Open AccessPublished:January 09, 2023DOI:https://doi.org/10.1016/j.xjidi.2023.100181

      1. ABSTRACT

      Documentation of change in number and appearance of pigmented cutaneous lesions over time is critical to early detection of skin cancers and may provide preliminary signals of efficacy in early phase therapeutic prevention trials for melanoma. Despite substantial progress in computer aided diagnosis of melanoma, automated methods to assess evolution of lesions are relatively undeveloped. This report describes the development and narrow validation of mathematical algorithms to register nevi between sequential digital photographs of large areas of skin and to align images for improved detection and quantification of changes. Serial posterior truncal photographs from a pre-existing database were processed and analyzed by the software, and the results were evaluated by a panel of clinicians using a separate XML-based application. The software had a high sensitivity for detection of cutaneous lesions as small as 2 mm. The software registered lesions accurately, with occasional errors at the edges of the images. In the 17-patient pilot study, use of the software enabled clinicians to identify new and/or enlarged lesions in 3 to 11 additional patients vs the unregistered images. Automated quantification of size change performed similarly to that of human raters. These results support the further development and broader validation of this technique.

      Abbreviations used:

      XML (Extensible Markup Language), A/DN (Atypical/Dysplastic Nevi), RGB (Red Green Blue), HIS (Hue Saturation Intensity), SIIM-ISIC (Society for Imaging Informatics in Medicine-International Skin Imaging Collaboration)

      2. INTRODUCTION

      Cutaneous melanoma may arise either de novo or from non-obligate precursor lesions such as atypical/dysplastic nevi (A/DN) (
      • Shain A.H.
      • Bastian B.C.
      From melanocytes to melanomas.
      ). Assessment of change in number and morphology of pigmented cutaneous lesions over time is used in many centers to improve the early detection of melanoma (
      • Banky J.P.
      • Kelly J.W.
      • English D.R.
      • Yeatman J.M.
      • Dowling J.P.
      Incidence of new and changed nevi and melanomas detected using baseline images and dermoscopy in patients at high risk for melanoma.
      ; Mar et al. 2017). Additionally, we and others have hypothesized that changes in the appearance of nevi may provide preliminary signals of efficacy in early phase therapeutic prevention (chemoprevention/immunoprevention) trials for melanoma (
      • Maguire W.F.
      • Kirkwood J.M.
      Developing agents for the therapeutic prevention of melanoma: Can the assessment of cutaneous precursor lesions help?.
      ), analogous to how changes in number of actinic keratoses provided a preliminary signal of the efficacy of nicotinamide in the prevention of nonmelanoma skin cancer (
      • Chen A.C.
      • Martin A.J.
      • Choy B.
      • Fernández-Peñas P.
      • Dalziell R.A.
      • McKenzie C.A.
      • et al.
      A phase 3 randomized trial of nicotinamide for skin-cancer chemoprevention.
      ;
      • Surjana D.
      • Halliday G.M.
      • Martin A.J.
      • Moloney F.J.
      • Damian D.L.
      Oral nicotinamide reduces actinic keratoses in phase II double-blinded randomized controlled trials.
      ). An objective, reproducible means of measuring changes in the appearance of cutaneous pigmented lesions will facilitate both clinical and research applications.
      Assessment of change of lesions in serial digital images by human observers is complicated by numerous factors (
      • Schneider S.L.
      • Kohli I.
      • Hamzavi I.H.
      • Council M.L.
      • Rossi A.M.
      • Ozog D.M.
      Emerging imaging technologies in dermatology: Part II: Applications and limitations.
      ). Factors relating to the images themselves may include differences in scale, body habitus, posture, and lighting between the serial photographs. Factors relating to the human evaluators include differences in provider experience, as well as time pressures posed by clinical workload and competing responsibilities. These issues contribute to limitations of human perception of subtle changes in multiple lesions simultaneously under suboptimal conditions. This is particularly relevant for nevi, a subset of which are dynamic but the majority of which are stable or show minor or no change over time when assessed by human reviewers (
      • Banky J.P.
      • Kelly J.W.
      • English D.R.
      • Yeatman J.M.
      • Dowling J.P.
      Incidence of new and changed nevi and melanomas detected using baseline images and dermoscopy in patients at high risk for melanoma.
      ). We propose that a computer vision approach can help mitigate these problems and produce more accurate assessment of change in cutaneous lesions.
      Despite the use of total body digital photography for at least 20 years to document the presence of cutaneous lesions (
      • Hornung A.
      • Steeb T.
      • Wessely A.
      • Brinker T.J.
      • Breakell T.
      • Erdmann M.
      • et al.
      The Value of Total Body Photography for the Early Detection of Melanoma: A Systematic Review.
      ), as well as substantial recent progress in computer-aided diagnosis of lesions in clinical images (
      • Dick V.
      • Sinz C.
      • Mittlböck M.
      • Kittler H.
      • Tschandl P.
      Accuracy of Computer-Aided Diagnosis of Melanoma: A Meta-analysis.
      ;

      Haenssle HA, Fink C, Toberer F, Winkler J, Stolz W, Deinlein T, et al. Man against machine reloaded: performance of a market-approved convolutional neural network in classifying a broad spectrum of skin lesions in comparison with 96 dermatologists working under less artificial conditions. Annals of Oncology. Elsevier; 2020;31(1):137–143

      ;
      • Tschandl P.
      • Codella N.
      • Akay B.N.
      • Argenziano G.
      • Braun R.P.
      • Cabo H.
      • et al.
      Comparison of the accuracy of human readers versus machine-learning algorithms for pigmented skin lesion classification: an open, web-based, international, diagnostic study.
      ), automated methods to characterize the evolution of skin lesions are still lacking. Automated image registration refers to the use of computational models to correlate features between two or more images. In many cases the process also includes transforming one or both of the images so that the features can be aligned for improved comparison. Numerous techniques for automated image registration have been developed over the past few decades and are of great interest in a variety of fields including remote sensing (for example detection of changes by orbiting satellites) (Tondewad and Dale 2020) and medical imaging (
      • Guan S.-Y.
      • Wang T.-M.
      • Meng C.
      • Wang J.-C.
      A Review of Point Feature Based Medical Image Registration.
      ). Registration and alignment of photographs of human skin pose a more challenging problem than registration in other domains, because skin requires accommodation of changes in weight, wrinkling, hair, scale, skin tone changes, lighting, translation, rotation, and warping concurrently.
      The purpose of this study was to develop and preliminarily validate a computer vision approach to facilitate the detection and quantification of changes in nevi in serial digital photographs.

      3. METHODS & DESCRIPTION

      Note: we have endeavored to conform to a published consensus checklist for evaluation of image-based artificial intelligence reports in dermatology (Daneshjou et al. 2022), to the extent that is possible given the proprietary nature of the DermaViz software.

      Description of DermaViz

      The registration approach utilized by the DermaViz software is composed of four main steps, for which the skin lesions are considered points in a coordinate system defined as the skin space. First, the technique computes a coarse estimate of scale and rotation of the points based on the log of potentially matched point differences. Second, it computes a low-resolution translation estimate leading to point matches and a nonlinear mapping. Third, the algorithm computes a high-resolution translation estimate over sub-image tiles of the entire image leading to improved point matches and nonlinear mapping. Lastly, the algorithm performs cross -checking of the point matches to eliminate false matches followed by re-iteration of these steps until all of the point matches are confirmed as accurately matched. Overall, if corresponding paired images have at least 50% common overlap in the recorded skin surface area, then the algorithms can accommodate resolutions ratios as disparate as 1:3 linear, 1:9 area, angular rotations of +/- 10 degrees along with mild key stoning and additionally translation shifts. Figure 1 shows registered and aligned areas of the posterior trunk of a patient photographed one year apart (a-b) as well as segmentation boundaries and measured changes (c). The aligned images can be toggled back and forth within the same frame to better visualize changes in individual lesions. Movies S1-2 show representative clinical images of two patients before and after processing with DermaViz, displayed in the toggle view. Movie S3 presents similar pre- and post-processing image pairs acquired with a mobile phone device.
      Figure thumbnail gr1
      Figure 1Demonstration of DermaViz software: (a) Initial timepoint (b) Subsequent timepoint after registration and alignment (c) Overlaid segmentation boundaries and example change quantification (d) Delineation of border (e) Delineation of diameter (f) Delineation of major and minor axes (g) Delineation of convex hull. Figures 1d-g are reproduced from Tahata et al. 2018. Images provided by Veytel.
      The segmentation approach to distinguish skin lesions from the background skin is an algorithm which converts the skin RGB image into three separate images for Hue, Saturation and Intensity (HSI) (
      • Blotta E.
      • Bouchet A.
      • Ballarin V.
      • Pastore J.
      Enhancement of medical images in HSI color space.
      ). Nevus filtering-thresholding algorithms for each of the hue, saturation and intensity domains are applied individually, with filters tuned for the statistics of the nevus and background skin regions respectively. The resulting three images are recombined to create a unified segmentation of the nevus / skin boundary.
      Once lesion segmentation is defined, size features are readily calculated for area, diameter, and perimeter. The basis of size and other measurements is illustrated in Figure 1d-g (reproduced from
      • Tahata S.
      • Singh S.V.
      • Lin Y.
      • Hahm E.-R.
      • Beumer J.H.
      • Christner S.M.
      • et al.
      Evaluation of biodistribution of sulforaphane after administration of oral broccoli sprout extract in melanoma patients with multiple atypical nevi.
      ) with the nevus shown in white and a rectangle enclosing the nevus that has the same major axis as the ellipse. The area is computed in square-mm using the pixels within the border. The diameter is the extent of the lesion in the direction of the major axis of an ellipse that has the same second moments as the nevus, measured in millimeters (mm). Perimeter is the measure of the distance around the border in mm. Table 1 details the classic “ABCDE” features of melanoma clinical presentation (
      • Glazer A.M.
      • Rigel D.S.
      • Winkelmann R.R.
      • Farberg A.S.
      Clinical Diagnosis of Skin Cancer: Enhancing Inspection and Early Recognition.
      ), as well as a general description of how they are quantified by the DermaViz software. Of these, the size-related measurements are the main focus of this study. It should be noted that in its current version, DermaViz does not classify lesions by type.
      Table 1ABCDE features and measurement by DermaViz
      QualityTypeDescription
      A

      Asymmetry
      ComponentMeasure of the asymmetrical components of the quadrants of the nevi.
      B

      Border Irregularity
      VariationMeasure of the difference of ratio of the nevus shape in each quadrant as compared to the symmetrical shape of the convex hull in each quadrant. Quadrants are defined by the major and minor axes as shown in Figure 1.
      C

      Color
      Color Saturation VariationMean color saturation across the four quadrants of the nevi, with the quadrants defined by the major and minor axes as shown in Figure 1.
      Color Intensity VariationMean color intensity across the four quadrants of the nevi, with the quadrants defined by the major and minor axes as shown in Figure 1.
      D

      Diameter (Size)
      AreaMeasure of pixels within the border
      DiameterExtent of the nevus in the direction of the major axis of an ellipse that has the same second moments as the nevus. Illustrated in Figure 1 with the nevus shown in white and a rectangle enclosing the nevus that has the same major axis as the ellipse.
      PerimeterDistance measured around the border
      E

      Evolution
      ChangeChange of any of the above features computed as difference between the original or older feature value to newer feature values.
      Percent ChangeChange normalized by the original feature value.

      Description of Validator application

      The Validator is an Extensible Markup Language (XML)-based image evaluation software application that allows clinicians to assess features of the DermaViz algorithms as well as to classify and characterize lesions detected by DermaViz. It is highly configurable, allowing the specific prompts and responses to be modified based on the objectives of the study for which it is used. In the current study there are three sections, as illustrated in Figure 2: a. Nevus manual identification (clinician clicks on the center of each nevus to verify accuracy of software lesion identification); b. Nevus pair description (clinician assesses the accuracy of lesion registration between timepoints and identifies the lesion at each timepoint via options on a drop-down menu); and c. Area change for center nevus (clinician enters subjective opinion of size change along a pre-specified scale via a drop-down menu).
      Figure thumbnail gr2
      Figure 2Demonstration of Validator application. (a) Nevus manual identification (b) Nevus pair description (c) Area change for center nevus. Images are screenshots from the clinician-facing portion of the Validator application, taken by a clinical reviewer.
      The Validator application runs on Mac and Windows computers with a separate installer for each. On software startup there is an option to log in as a known user (named in advance in the software) or as an unknown user (delineated by a user number). Upon completion of the questions, the responses are electronically submitted to Veytel for aggregation and analysis.

      Characteristics of posterior truncal photographs used in the pilot study

      Serial posterior truncal photographs from patients with multiple atypical nevi and a history of melanoma were obtained from a pre-existing image and nevus biobanking protocol database at UPMC Hillman Cancer Center. The original images were taken under protocol UPCI 96-099 (IRB # REN18100233 / IRB970186 approved by University of Pittsburgh Institutional Review Board) using a single Nikon D700 camera using a standardized background and automatic focus and exposure settings. They were stored using Philips iSITE PACS (Koninklijke Philips NV, Eindhoven, NL). Patients provided written, informed consent for use of the de-identified images in future research. The images were stripped of all identifiers other than a study-specific identification code and sent to Veytel. Preprocessing consisted only of cropping some images to remove the arms; images were then analyzed with DermaViz software.
      For inclusion in the study, the patients’ images had to meet the following criteria (1) number of nevi / lesions imaged in the photos adequate for registration (roughly greater than 20); (2) moderate posture changes (not severe); (3) similar cropping (i.e., including / excluding shoulders); (4) interval between imaging dates less than 6 years.
      For the mobile photographs included in the study, images were acquired using an iPhone 11 (Apple, Cupertino, CA). A volunteer with multiple cutaneous nevi was recruited by Veytel and signed consent forms for both photography and publication of images. Neither the recruitment of the volunteer nor the photography were connected with clinical care, however the processed images were provided to the patient to share with their healthcare providers.

      Change assessment in posterior truncal images

      Clinicians were asked to review paired serial posterior truncal images presented as Microsoft PowerPoint slides that could be toggled back and forth between dates. The clinicians first reviewed the set of unmodified photographs, and then in a separate session reviewed the same images that had been registered and aligned by DermaViz. There was no size threshold for lesions to be included in this portion of the study.

      Assessment of sensitivity

      Clinicians were shown randomly chosen subfields of the posterior truncal images in the Validator application and asked to click on all pigmented lesions of interest in the images, which marked them with crosshairs and recorded the corresponding coordinates (Figure 2a). The software then determined whether the crosshairs were within the segmentation boundaries of a lesion that had been identified by DermaViz. This allowed identification of true positives (defined as lesions for which both clinicians and DermaViz identified a nevus), false negatives (defined as lesions for which clinicians but not DermaViz identified a distinct nevus), and non-nevus lesions (defined as lesions identified by DermaViz that were not identified as nevi by clinicians). Sensitivity was calculated as (true positives)/(true positives + false negatives).

      Assessment of the accuracy of registration and nevus characterization

      For the first method of assessing the quality of registration, reviewers who completed the “change assessment in posterior truncal images” exercise with aligned PowerPoint slides were asked to comment on whether each image pair had approximately > 90% of the lesions accurately registered, as well as to comment subjectively on any issues with registration.
      For the second method of assessing the quality of registration, a subset of lesions identified by DermaViz were selected for clinician review based on the following criteria: 1. All lesions over 4 mm in diameter; 2. Lesions with visibly apparent changes in paired images as determined by staff at Veytel; 3. Largest remaining lesion in each quadrant of the posterior truncal image pair if lesions are present, excluding nevi near the edge of the edge of the images.
      Clinicians were shown image pairs of randomly chosen lesions photographed on different dates in the Validator application and asked whether the serial image pairs were registered (Figure 2b). “Registered” was defined in the following way: if there is change, at least 90% of the area of the smaller of the nevus pair is contained within the larger of the lesion pair. If there is not change, then the lesion pairs overlap by at least 90%. When the lesion pair is not correctly registered, the earlier and later date lesion do not align, ie there is less than 90% overlap of the smaller lesion with the larger lesion. Two visualization options were offered: a view with the two timepoints shown side-by-side; and a toggle view where the aligned images could be toggled back and forth between timepoints. This section of the validator also allowed clinicians to classify the lesions by type from a drop-down menu.

      Assessment of the accuracy of size change

      The first method of assessing the accuracy of size change used the Validator application, for which a subset of lesions was selected for further analysis according to the three criteria specified in the previous section. Clinicians were shown sequential zoomed-in views of isolated lesions, which were chosen at random (Figure 2c). Two visualization options were offered: a view with the two timepoints shown side-by-side; and a toggle view where the overlaid images could be toggled back and forth between timepoints. Clinicians were asked to approximately categorize the change observed into specific bins: disappeared (100%), substantial decrease (> 30%), moderate decrease (15-30%), no/minimal change (< 15%), moderate increase (15-30%), and substantial increase (> 30%). The results were compared both between observers and to the DermaViz quantitative assessments of diameter and area.

      Statistical methods

      For the number of new/increased lesions, a linear mixed model was used to study its association with image registration. For the variable DETECTED, which is defined as 1 if a patient had new/increased lesions detected and as 0 otherwise, a linear mixed effect logistic regression model was used to study its association with image registration. In both of the above two mixed models, the registration (=1 for a registered image, =0 for an unregistered image) was a fixed effect, and patient and clinician were two random effects. The first model was analyzed with Proc Mixed in SAS software (SAS Institute, Cary, NC). The second model was analyzed with SAS Proc GLIMMIX.
      For calculation of confidence intervals for sensitivity and false negative rate, we used the Clopper-Pearson interval, which is based on the cumulative probability of the binomial distribution (
      • Clopper C.J.
      • Pearson E.S.
      The use of confidence or fiducial limits illustrated in the case of the binomial.
      ).
      For assessment of size change, Cohen’s kappa was used to compute kappa values between each individual rater (Humans 1-3 and DermaViz), using both linear and quadratic weights, as well as 95% confidence intervals for each comparison. Analysis was performed in STATA (StataCorp, College Station, TX) using the “kappaetc” program written by Daniel Klein and available through the Boston College Statistical Software Components (SSC) archive.

      4. RESULTS

      Features of the pilot study

      Image pairs of the posterior trunk from 24 patients were obtained from the archival image database, of whom 7 were excluded from analysis by DermaViz due to not meeting the criteria for inclusion listed in the Methods. The most common reasons for exclusion were image pairs with too few features to register, images with poor acquisition such as uneven/poor cropping, and duration > 6 years between images.
      Of the patients whose photographs were analyzed, the mean age at the time of the first photograph was 38.4 years old (range 8.8-79.0, standard deviation 16.8 years). Seven patients were men and 10 were women. Recorded Fitzpatrick skin types were Fitzpatrick 1 (4 patients), Fitzpatrick 1-2 (2 patients), Fitzpatrick 2 (8 patients), Fitzpatrick 2-3 (1 patient), and unknown (2 patients). Among patients for whom historical information was available, 14 carried a prior diagnosis of melanoma. The remaining 3 patients did not have a prior histopathologic diagnosis available. One patient had 2 pairs of photographs, so there were 18 total image pairs included in the analysis. The mean time between paired photographs was 3.4 years (range 0.5-7.2 years, standard deviation 1.4 years).

      Change assessment in posterior truncal images

      To assess whether registration and alignment by the software improved clinicians’ ability to detect change, 3 practicing dermatologists and 2 medical oncologists involved in the care of melanoma patients reviewed the paired serial full-back images and enumerated number of pigmented lesion pairs in several categories: decreased, disappeared, surgically removed, increased, or appeared. To reduce the potential for bias due to carry-over effects, the images in each set (unregistered and registered) were presented in random order, and each set of images was presented during different viewing sessions with the unregistered images shown first. The mean delay between viewing the first and second sets of images was 14 days (range 0.167–40 days, standard deviation 17 days).
      Clinician use of DermaViz increased the number of lesions detected in several categories: total changed lesions, new/increased lesions, decreased/disappeared lesions, surgically removed lesions (Figure 3a), and image pairs with new/increased lesions detected (Figure 3b). The increase in detection of total changed lesions (average 1.7-fold increase) was primarily driven by a 1.8-fold average increase in detection of new or increased lesions (Table 2). The changes in decreased/disappeared lesions and surgically removed lesions were less consistent and of lower magnitude.
      Figure thumbnail gr3
      Figure 3Effects of DermaViz on detection of change by clinicians. (a) Nevus change detection by category (b) New/Increased nevi detected (c) Image pairs with new/increasing nevi detected. Error bars represent standard error of the mean. Derm = dermatologist; Med Onc = medical oncologist.
      Table 2Numerical results of change assessment exercise.
      ProviderImage pair typeTime between viewing registered and unregistered images (days)Number of total changes detectedNumber of new/increased lesions detectedNumber of decreased/disappeared lesions detectedNumber of surgically removed lesions detectedNumber of image pairs with new/increased lesions detected
      Dermatologist 1Unregistered21158117301114
      Registered244203251618
      Dermatologist 2Unregistered107546171216
      Registered8257111415
      Dermatologist 3Unregistered40482861413
      Registered755610916
      Medical Oncologist 1Unregistered1102352
      Registered37238613
      Medical Oncologist 2Unregistered0.1678461111214
      Registered183121461617
      Total MeanUnregistered14.475.050.813.410.811.8
      Registered124.292.020.012.215.8
      Total standard deviationUnregistered16.654.643.010.73.45.6
      Registered86.071.516.04.51.9
      Fold difference registered/unregistered1.71.81.51.11.3
      DermaViz improved the detection of new or increased lesions by all providers (Figure 3c), despite differences in the specialty of provider, amount of time elapsed between viewing the unregistered and registered images, and baseline number of images detected in each category. Complete numerical results are provided in Table 2.
      Registration produced statistically significant improvements in the identification of new and increased lesions (p < 0.001), and in the identification of image pairs with changes detected (p < 0.001) vs unregistered images, as assessed by linear mixed effects models. The odds ratio for changes detected between registered and unregistered images was 4.85 (95% confidence interval 2.06-11.39).

      Assessment of sensitivity and characterization of detected lesions

      To evaluate the accuracy of DermaViz software to identify nevi and other cutaneous lesions, we compared the results of lesion identification by the software with clinician review of 46 randomly selected subfields of the posterior truncal images by 2 medical oncologists and 1 advanced practice provider, all of whom were involved in the care of melanoma patients as detailed in the methods. Consensus of 3/3 clinicians was considered a gold standard of lesion identification in this study. Of note, there were no disagreements between clinicians regarding nevus identification. Figure 4a and 4b show representative images of common and atypical nevi, respectively.
      Figure thumbnail gr4
      Figure 4Spectrum of lesions detected by DermaViz. (a) Common melanocytic nevus. (b) Atypical melanocytic nevus (c) Adjacent nevi counted as a single nevus (d) Nonmelanocytic lesion (seborrheic keratosis) (e) Indeterminate lesion (likely artifact such as dust on lens). Images are screenshots from the clinician-facing portion of the Validator application, taken by a clinical reviewer.
      For lesions over 2mm where 3/3 clinicians identified a nevus, DermaViz agreed for 67/69 lesions. This corresponds to a false negative rate of 2.90% (exact 95% confidence interval 0.8%-7.5%) and a sensitivity of 97.1% (exact 95% confidence interval 92.5%-99.2%). The two nevi “missed” by DermaViz were both small nevi that were counted together with immediately adjacent larger nevi (Figure 4c).
      As noted above, the current version of DermaViz is not intended to classify lesions by type, however this information can be added by clinicians using the Validator application. Of the 72 lesions over 2mm identified by DermaViz in the nevus centroid exercise, 67 (93%) were identified as melanocytic nevi by 3/3 clinicians. In the nevus size exercise, in which the lesions were selected via different criteria as noted in the Methods, a lower percentage of lesions were found to be melanocytic (average 66/97 lesions or 68%).
      Non-nevus lesions detected by DermaViz were frequently classified by clinicians as seborrheic keratoses (example shown in Figure 4d) but other non-melanocytic structures were identified by the software as well, including occasional suspected artifacts including possible dust on the camera lens (Figure 4e).

      Assessment of the accuracy of registration

      Two methods were used to assess whether DermaViz software had successfully registered nevi between timepoints for individual patients. The first method involved the change detection exercise involving aligned posterior truncal photographs displayed as sequential PowerPoint slides. The five clinician reviewers who participated in this exercise determined that on average, 17/18 images (range 15-18) had greater than 90% of lesions correctly aligned. The second method involved comparison in the Validator application of 97 paired fields, which included a variety of lesion types including those with increasing, decreasing, appearing, and disappearing lesions. Of the 97 paired fields, 87 clearly had lesions identified in both images, while the remaining 10 had lesions identified that either appeared or disappeared. The three clinician reviewers who completed this exercise determined that all 87 images included in this analysis that had paired lesions in the subsequent image had been correctly registered. Of note, lesions on the edges of the images had been specifically excluded for this analysis.
      Despite most lesions being correctly registered, there were still occasional errors. Movie S1 illustrates a typical image pair where most lesions are correctly registered but there are a limited number of nevi on the edges of the image that are incorrectly registered and not properly aligned.

      Assessment of the accuracy of size change

      To compare the assessment of size change by humans to the DermaViz quantification, three clinicians (2 medical oncologists and 1 advanced practice provider) were shown sequential zoomed-in views of lesion pairs in the Validator interface, which were selected as described in the Methods section. Clinicians were asked to characterize changes in the size of the lesions on an ordinal scale. Interrater reliability between the clinicians and each other, and between each clinician and DermaViz, were computed using Cohen’s kappa. Data are presented as weighted kappas. We feel that quadratic weighting relates more to the current analysis, since it gives harsher penalties to larger disagreements; and also because it produces similar results to the Fleiss intraclass correlation, another commonly-used method of interrater reliability (

      Fleiss JL, Cohen J. The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability. Educ Psychol Meas. SAGE Publications Inc; 1973;33(3):613–619

      ). However, we have reported both linear and quadratic weights because of varying opinions in the literature over which weighting system is preferable (
      • Vanbelle S.
      A New Interpretation of the Weighted Kappa Coefficients.
      ).
      As shown in Figure 5, clinicians showed moderate (with linear weights) to good (with quadratic weights) agreement with each other. DermaViz also showed moderate (with linear weights) to good (with quadratic weights) agreement with the clinicians. The weighted kappas were slightly lower for DermaViz vs the human raters, however the 95% confidence intervals overlap such that the difference was not significant for any rater pair. The numerical kappa values between raters, as well as the individual 95% confidence intervals, are shown in Table 3, Table 4.
      Figure thumbnail gr5
      Figure 5Size change assessment by DermaViz compared with that of human observers. Bars represent weighted kappas between all human observers (H1, H2, and H3), and between each human observer and DermaViz (DV).
      Table 3Linear weighted Cohen’s kappa between each pair of two raters for size change exercise, +/- 95% confidence intervals. H1=Human 1, DV = DermaViz.
      RaterH1H2H3DV
      H1-0.7636 (0.651-0.8522)0.6819 (0.5862-0.7777)0.6924 (0.5870-0.7978
      H2--0.7095 (0.6221-0.7970)0.63 (0.5195-0.7404)
      H3---0.6106 (0.4991-0.7221)
      DV----
      Table 4Quadratic weighted Cohen’s kappa between each pair of two raters for size change exercise, +/- 95% confidence intervals. H1 = Human 1, DV = DermaViz.
      RaterH1H2H3DV
      H1-0.8958 (0.8415-0.9500)0.8542 (0.7882-0.9202)0.8553 (0.7840-0.9267)
      H2--0.8708 (0.8131-0.9284)0.8322 (0.7558-0.9087)
      H3---0.8030 (0.7172-0.8888)
      DV----

      5. DISCUSSION AND POTENTIAL APPLICATIONS

      In this study, DermaViz automated image registration facilitated detection and quantification of changes in size and number of pigmented lesions of interest in sequential digital photographs by melanoma clinicians. The software helped clinicians to identify numerous changes that were missed in the original unregistered images. Importantly, because the DermaViz approach utilizes photographs of large areas of skin, it potentially offers an important advantage over the current clinical practice of change assessment through serial dermoscopy of single pigmented lesions (
      • Morris J.B.
      • Alfonso S.V.
      • Hernandez N.
      • Fernández M.I.
      Use of and intentions to use dermoscopy among physicians in the United States.
      ). Pigmented lesions of concern must be pre-specified by physicians before they can be evaluated via dermoscopy; such focused assessment will miss changes in lesions that were not specifically selected for baseline imaging (
      • Fuller S.R.
      • Bowen G.M.
      • Tanner B.
      • Florell S.R.
      • Grossman D.
      Digital dermoscopic monitoring of atypical nevi in patients at risk for melanoma.
      ). DermaViz inherently tracks multiple pigmented lesions contained in the photographs and computes accurate relative change between sequential photograph dates. A promising combination strategy might be an initial step of automated image analysis of digital camera photographs, followed by focused dermoscopy of concerning lesions.
      There have been relatively few prior attempts at registration of skin lesions in digital photographs of large areas of skin, and our experience with DermaViz compares favorably to these attempts. An early approach for registration represented pigmented lesions as points on a lesion map, identified initial matching lesion pairs in the sequential images, selected four matching image pairs near the image corners using an iterative method designed to reduce mismatches, and then transformed one image to match the lesion map of the other image using the four selected points (
      • Mcgregor B.
      Automatic registration of images of pigmented skin lesions.
      ). Results were promising, yet the testing used data from the same date, and while it accounted for some posture changes, it did little to handle lighting or scale variations. Another approach to registration included alignment using structured geometric and tensor-based models, yet this approach was not shown to be robust to changes in posture or scale (
      • Mirzaalian H.
      • Lee T.K.
      • Hamarneh G.
      Skin lesion tracking using structured graphical models.
      ). In the Canfield approach (
      • Korotkov K.
      • Quintana J.
      • Puig S.
      • Malvehy J.
      • Garcia R.
      A new total body scanning system for automatic change detection in multiple pigmented skin lesions.
      ), the lesion registration and flickering is robust yet was demonstrated only with specific nevus pairs as opposed to detecting change in larger body regions of the body such as the posterior trunk, leg, or arm. Canfield and other approaches have used 3-dimensional total body photography (
      • Hornung A.
      • Steeb T.
      • Wessely A.
      • Brinker T.J.
      • Breakell T.
      • Erdmann M.
      • et al.
      The Value of Total Body Photography for the Early Detection of Melanoma: A Systematic Review.
      ), which minimizes variability in lighting and posture because of constraints on the patient. The Fotofinder technology includes advanced image acquisition hardware and lesion classification (
      • Del Rosario F.
      • Farahi J.M.
      • Drendel J.
      • Buntinx-Krieg T.
      • Caravaglio J.
      • Domozych R.
      • et al.
      Performance of a computer-aided digital dermoscopic image analyzer for melanoma detection in 1,076 pigmented skin lesion biopsies.
      ;
      • Levy J.L.
      • Trelles M.A.
      • Levy A.
      • Besson R.
      Photography in dermatology: comparison between slides and digital imaging.
      ), however we found no publication noting that this technology performs automated registration and detection of change. For both Canfield and Fotofinder approaches, the use of expensive, specialized imaging hardware may limit their use to specific clinical settings. The approach used by DermaViz is one that may affordably be implemented by a variety of dermatology and potentially even primary care settings. Furthermore DermaViz could theoretically be used to retrospectively compare archival digital camera images, whereas approaches requiring advanced image acquisition devices are limited by the relatively recent introduction of these devices.
      The DermaViz software can theoretically be used to improve the comparison of images obtained with mobile devices such as smartphones and tablets, for which camera capabilities have improved markedly in recent years. Movie S3 demonstrates successful registration and alignment of images obtained by an iPhone 11. There has been considerable interest in the use of smartphone applications to facilitate computer-aided diagnosis and monitoring of suspicious skin lesions, although results generated by the software have so far not been shown to be of clinical utility (

      Freeman K, Dinnes J, Chuchu N, Takwoingi Y, Bayliss SE, Matin RN, et al. Algorithm based smartphone apps to assess risk of skin cancer in adults: systematic review of diagnostic accuracy studies. BMJ. British Medical Journal Publishing Group; 2020;368:m127

      ;
      • Matin R.N.
      • Dinnes J.
      AI-based smartphone apps for risk assessment of skin cancer need more evaluation and better regulation.
      ). A recent study demonstrated that parameters of commercial digital cameras and smartphones could be modified to increase accuracy of the images, generally by disabling some automatic settings that were intended to produce more visually appealing images to consumers but can reduce accuracy (

      Dugonik B, Dugonik A, Marovt M, Golob M. Image Quality Assessment of Digital Image Capturing Devices for Melanoma Detection. Applied Sciences. Multidisciplinary Digital Publishing Institute; 2020;10(8):2876

      ). For optimal results with DermaViz analysis of smartphone images, it will be important to specify protocols for image acquisition to promote accuracy and particularly consistency between images acquired over time. This process may be aided by further progress in developing standards for the imaging of skin lesions (
      • Finnane A.
      • Curiel-Lewandrowski C.
      • Wimberley G.
      • Caffery L.
      • Katragadda C.
      • Halpern A.
      • et al.
      Proposed Technical Guidelines for the Acquisition of Clinical Images of Skin-Related Conditions.
      ). Given the anticipated need for at least some standardization in image acquisition, we anticipate that smartphone images will need to be taken in clinical settings with appropriate training, and we do not anticipate that patient-obtained smartphone photographs will generally be suitable for use with DermaViz. The main implication of using DermaViz with smartphone photographs is that it would greatly expand the number and type of physician offices that could use the software.
      Several images in the current study could not be accurately registered because of issues with suboptimal quality and heterogeneity of the archival images; however it should be noted that the images were not acquired for this purpose and did not have consistent acquisition parameters. We expect that prospective photography using even simple uniform acquisition parameters will result in a higher percent of images that can be registered by DermaViz. Although there is no requirement for a particular resolution of camera/image needed to use DermaViz, we expect that high resolution and in-focus images will provide additional detail that will increase the number of image pairs that can be successfully registered, as well as the accuracy of the registration.
      A notable feature of the current study was the use of sequential images from patients that were taken as part of an image and nevus biobanking protocol at our institution. This differs from several other studies that used artificial means of generating changes in size and number of cutaneous lesions prior to analysis, either by drawing them on the skin (
      • Guido N.
      • Hagstrom E.
      • Ibler E.
      • Carneiro C.
      • Orrell K.
      • Kelm R.
      • et al.
      A novel total body digital photography smartphone application designed to detect and monitor skin lesions: a pilot study.
      ) or by digitally modifying the original images to introduce the changes (
      • Navarro F.
      • Escudero-Viñolo M.
      • Bescós J.
      Accurate Segmentation and Registration of Skin Lesion Images to Evaluate Lesion Change.
      ). The use of artificial nevi and/or digital changes to nevi may not reflect a real world situation and may introduce artifacts and/or biases due to assumptions made when creating or modifying the lesions. The use of sequential patient images is more relevant to real clinical situations, but it also makes it more challenging to identify an appropriate gold standard and/or define a ground truth with which to compare the results from DermaViz. A common gold standard method that has been used to evaluate quality of segmentation in unmodified clinical images is the Society for Imaging Informatics in Medicine-International Skin Imaging Collaboration (SIIM-ISIC) Melanoma Classification challenge (
      • Rotemberg V.
      • Kurtansky N.
      • Betz-Stablein B.
      • Caffery L.
      • Chousakos E.
      • Codella N.
      • et al.
      A patient-centric dataset of images and metadata for identifying melanomas using clinical context.
      ), which has been used to evaluate segmentation and classification capabilities of numerous other image analysis tools. We did not use this for the current study because our images are camera photographs of the whole posterior trunk, whereas the images in the SIIM-ISIC are generally dermatoscopic images of single skin lesions. As previously noted, the DermaViz method enables tracking of any nevus that changes in the photograph, including those that were not identified as atypical on earlier dates. Furthermore the assessment technique used by DermaViz is inherently different than techniques focused on analysis of single lesions, in that the co-registration of multiple lesions contributes to higher accuracy change assessment than if DermaViz was used to analyze individual nevi. Therefore, the segmentation of individually chosen lesions is not fully indicative of DermaViz’s capabilities. It could be argued that the use of clinical observers as was done here is not an ideal gold standard either, particularly since the automated analysis can leverage additional features of the image such as altered color spectra that are not visible to human observers. The small discrepancies between human observers and DermaViz may reflect deficiencies in human perception or deficiencies of the imaging program, and it is difficult to separate these two possibilities currently. We plan to address this in further studies, for example by presenting the human clinicians with modified images that represent the optimal color spectrum that DermaViz uses to detect change.
      The conclusions of the current study may have been affected by limitations of the image dataset used and the DermaViz software itself, both of which we hope to address in the future. Regarding limitations of the image dataset, these historical photographs were taken by research assistants. There were variations in background, lighting, focus, and exposure, and it is conceivable that changes in imaging parameters might contribute to some of the observed changes. The images did not consistently incorporate a measurement ruler and used no form of calibration for color or lighting. There was a wide range of follow-up intervals between images. Finally, the image dataset comprised a small number of patients at a single institution. To address these potential issues in future studies, we have designed and are now including a new measurement tape including a ruler and color balance feature in every image for improved characterization of size, border, and color changes. At the UPMC Hillman Melanoma Program, photographs have for the past two years been taken by a single professional photographer with a high-resolution 12-bit digital camera and standardized lighting and formatting. Prospective studies will obtain images at consistent, pre-specified time intervals and will ideally include involvement with multiple clinical sites to improve external validity. It will be important to include many more patients, including patients with diverse skin types. Regarding the DermaViz software itself, there are four primary limitations of the current version of DermaViz. First, the automated algorithms do not yet determine lesion type, although clinicians are readily able to input this information using the Validator interface. Future versions of DermaViz could include a computer-aided diagnosis feature, and/or the current software could be combined with one of many existing computer-aided diagnosis programs. Second, normalization of color and scale between images using the ruler/color balance tape has not yet been validated because we have only recently started using the updated tapes. Third, the technology has so far not been extensively used to characterize lesions on curved surfaces such as limbs, and this will need to be explored more in future studies. Finally, DermaViz’s metrics to quantitate border irregularity, asymmetry, and color will need to be validated before their accuracy/clinical relevance can be demonstrated. Of note regarding the last point, we expect that the current version of DermaViz, which primarily measures changes in size and number of lesions, will still be useful for a variety of applications even before the additional quantitation of borders, color, and asymmetry has been validated.
      The intended use of DermaViz at this point is in additional research studies in clinical office settings. These subsequent studies are needed to validate the software more broadly for both clinical and research uses. In terms of clinical use, a critical question that will eventually need to be addressed prospectively is whether the increased ability to detect changes noted in the current study results in differences in patient care such as different rates of biopsy and ultimately different rates of skin lesion diagnosis. This type of impact analysis could be achieved through a larger prospective study that compares the care of patients whose providers utilized DermaViz with a separate group of patients whose providers used traditional methods of monitoring skin lesions. Future studies should consider the timecourse of the changes, since more rapid changes would generally be more concerning. It will be important to also assess possible effects on overdiagnosis and physician fatigue, which are risks of this approach. In terms of research use, further validation of DermaViz will involve using the software to analyze changes in size and number (and later color, border, etc) between patient groups that are prospectively exposed to different treatment conditions to demonstrate whether changes in nevus morphology can be detected between those groups. After discussion with numerous melanoma experts, we have hypothesized that morphologic changes in size and number of A/DN would be the most likely useful surrogate for the efficacy of therapeutic prevention agents (chemoprevention/immunoprevention) (
      • Maguire W.F.
      • Kirkwood J.M.
      Developing agents for the therapeutic prevention of melanoma: Can the assessment of cutaneous precursor lesions help?.
      ), although it remains to be demonstrated prospectively whether any changes observed correlate with other biomarkers of melanoma development or with meaningful clinical outcomes. To this end, we intend to use DermaViz in a planned Phase II study of one year of treatment with the broccoli-derived agent sulforaphane vs placebo in patients with multiple atypical nevi and a history of melanoma (EA6201).
      This publication of the description and narrow validation of DermaViz software may raise awareness of the technology in the dermatology and oncology community, which we hope will facilitate the prospective studies that will ultimately be needed to more fully develop and validate this technology.

      6. DATA AVAILABILITY STATEMENT

      The deidentified posterior truncal images used in this study have been made publicly available and can be found at https://doi.org/10.34970/630662, hosted at the ISIC Archive (

      Kirkwood JM. Longitudinal overview images of posterior trunks [Internet]. ISIC Archive; 2022 [cited 2022 Nov 3]. Available from: https://api.isic-archive.com/collections/168/

      ). Requests for additional data related to this study should be addressed to the corresponding author.

      8. CONFLICT OF INTEREST

      PHH, CMD, MH, KJM, and EKH are employees of Veytel, LLC. At the time of the study, CSB, RJ, and GF were interns for Veytel, LLC. LKF is an investigator for DermTech Inc and Skin Analytics and is a consultant for DermTech Inc. JMK reports Consulting for Applied Clinical Intelligence, LLC, Amgen, Inc., Ankyra Therapeutics, Axio Research / Instil Bio, Becker Pharmaceutical Consulting, Bristol Myers Squibb, Checkmate Pharmaceuticals, DermTech, Fenix Group International, Harbour BioMed, Immunocore LLC, Intellisphere, LLC / Cancer Network, Iovance Biotherapeutics, IQVIA, Istari Oncology, Merck, Millennium Pharmaceuticals / Takeda Pharmaceutical, Natera Inc., Novartis Pharmaceuticals, OncoCyte Corporation, OncoSec Medical Inc., Pfizer, Replimune, Scopus BioPharma, SR One Capital Management, and Grant Support to Institution from Amgen, Inc., Bristol Myers Squibb, Castle Biosciences, Inc., Checkmate Pharmaceuticals, Harbour BioMed, Immvira Pharma Co., Immunocore LLC, Iovance Biotherapeutics, Merck, Novartis Pharmaceuticals, Schering-Plough, Takeda, and Verastem, Inc.

      10. AUTHOR CONTRIBUTION STATEMENT

      Contribution Author (initials)
      Conceptualization WFM, PHH, CMD, MH, KJM, EKH, JMK
      Data Curation WFM, KJM, EKH
      Formal Analysis WFM, PHH, CMD, MH, HW, KJM, EKH
      Funding Acquisition EKH, JMK
      Investigation CS, JMK
      Methodology WFM, CMD, HW, KJM, EKH, JMK
      Project Administration WFM, CS, EKH
      Resources EKH, JMK
      Software PHH, CMD, MH, CSB, RJ, GF, KJM, EKH
      Supervision EKH, KJM, JMK
      Validation WFM, CM, LKF, GP, JA, JMK
      Visualization WFM, EKH
      Writing- Original Draft Preparation WFM, EKH, JMK
      Writing- Review and Editing WFM, PHH, CMD, MH, CSB, RJ, GF, CM, CS, LKF, GP, JA, HW, KJM, EKH, JMK

      Uncited reference

      • Daneshjou R.
      • Barata C.
      • Betz-Stablein B.
      • Celebi M.E.
      • Codella N.
      • Combalia M.
      • et al.
      Checklist for evaluation of image-based artificial intelligence reports in dermatology: CLEAR Derm consensus guidelines from the International Skin Imaging Collaboration Artificial Intelligence Working Group.
      ,
      • Mar V.J.
      • Chamberlain A.J.
      • Kelly J.W.
      • Murray W.K.
      • Thompson J.F.
      Clinical practice guidelines for the diagnosis and management of melanoma: melanomas that lack classical clinical features.
      ,
      • MsPS Tondewad
      • Dale MsMP.
      Remote Sensing Image Registration Methodology: Review and Discussion.
      .

      9. ACKNOWLEDGMENTS

      This research was supported in part by philanthropic support from Making Melanoma History and the Building Bridges effort of the Melanoma Center, as well as from the Veytel Research Fund.

      Supplementary material

      11. REFERENCES

        • Banky J.P.
        • Kelly J.W.
        • English D.R.
        • Yeatman J.M.
        • Dowling J.P.
        Incidence of new and changed nevi and melanomas detected using baseline images and dermoscopy in patients at high risk for melanoma.
        Arch. Dermatol. 2005; 141: 998-1006
        • Blotta E.
        • Bouchet A.
        • Ballarin V.
        • Pastore J.
        Enhancement of medical images in HSI color space.
        J. Phys.: Conf. Ser. IOP Publishing. 2011; 332012041
        • Chen A.C.
        • Martin A.J.
        • Choy B.
        • Fernández-Peñas P.
        • Dalziell R.A.
        • McKenzie C.A.
        • et al.
        A phase 3 randomized trial of nicotinamide for skin-cancer chemoprevention.
        N. Engl. J. Med. 2015; 373: 1618-1626
        • Clopper C.J.
        • Pearson E.S.
        The use of confidence or fiducial limits illustrated in the case of the binomial.
        Biometrika. 1934; 26: 404-413
        • Daneshjou R.
        • Barata C.
        • Betz-Stablein B.
        • Celebi M.E.
        • Codella N.
        • Combalia M.
        • et al.
        Checklist for evaluation of image-based artificial intelligence reports in dermatology: CLEAR Derm consensus guidelines from the International Skin Imaging Collaboration Artificial Intelligence Working Group.
        JAMA Dermatol. 2022; 158: 90-96
        • Del Rosario F.
        • Farahi J.M.
        • Drendel J.
        • Buntinx-Krieg T.
        • Caravaglio J.
        • Domozych R.
        • et al.
        Performance of a computer-aided digital dermoscopic image analyzer for melanoma detection in 1,076 pigmented skin lesion biopsies.
        J Am Acad Dermatol. 2018; 78 (e6): 927-934
        • Dick V.
        • Sinz C.
        • Mittlböck M.
        • Kittler H.
        • Tschandl P.
        Accuracy of Computer-Aided Diagnosis of Melanoma: A Meta-analysis.
        JAMA Dermatology. 2019; 155: 1291-1299
      1. Dugonik B, Dugonik A, Marovt M, Golob M. Image Quality Assessment of Digital Image Capturing Devices for Melanoma Detection. Applied Sciences. Multidisciplinary Digital Publishing Institute; 2020;10(8):2876

        • Finnane A.
        • Curiel-Lewandrowski C.
        • Wimberley G.
        • Caffery L.
        • Katragadda C.
        • Halpern A.
        • et al.
        Proposed Technical Guidelines for the Acquisition of Clinical Images of Skin-Related Conditions.
        JAMA Dermatology. 2017; 153: 453-457
      2. Fleiss JL, Cohen J. The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability. Educ Psychol Meas. SAGE Publications Inc; 1973;33(3):613–619

      3. Freeman K, Dinnes J, Chuchu N, Takwoingi Y, Bayliss SE, Matin RN, et al. Algorithm based smartphone apps to assess risk of skin cancer in adults: systematic review of diagnostic accuracy studies. BMJ. British Medical Journal Publishing Group; 2020;368:m127

        • Fuller S.R.
        • Bowen G.M.
        • Tanner B.
        • Florell S.R.
        • Grossman D.
        Digital dermoscopic monitoring of atypical nevi in patients at risk for melanoma.
        Dermatol Surg. 2007; 33: 1198-1206
        • Glazer A.M.
        • Rigel D.S.
        • Winkelmann R.R.
        • Farberg A.S.
        Clinical Diagnosis of Skin Cancer: Enhancing Inspection and Early Recognition.
        Dermatol Clin. 2017; 35: 409-416
        • Guan S.-Y.
        • Wang T.-M.
        • Meng C.
        • Wang J.-C.
        A Review of Point Feature Based Medical Image Registration.
        Chinese Journal of Mechanical Engineering. 2018; 31: 76
        • Guido N.
        • Hagstrom E.
        • Ibler E.
        • Carneiro C.
        • Orrell K.
        • Kelm R.
        • et al.
        A novel total body digital photography smartphone application designed to detect and monitor skin lesions: a pilot study.
        Journal of Surgical Dermatology. 2018; 3
      4. Haenssle HA, Fink C, Toberer F, Winkler J, Stolz W, Deinlein T, et al. Man against machine reloaded: performance of a market-approved convolutional neural network in classifying a broad spectrum of skin lesions in comparison with 96 dermatologists working under less artificial conditions. Annals of Oncology. Elsevier; 2020;31(1):137–143

        • Hornung A.
        • Steeb T.
        • Wessely A.
        • Brinker T.J.
        • Breakell T.
        • Erdmann M.
        • et al.
        The Value of Total Body Photography for the Early Detection of Melanoma: A Systematic Review.
        Int J Environ Res Public Health. 2021; 18: 1726
      5. Kirkwood JM. Longitudinal overview images of posterior trunks [Internet]. ISIC Archive; 2022 [cited 2022 Nov 3]. Available from: https://api.isic-archive.com/collections/168/

        • Korotkov K.
        • Quintana J.
        • Puig S.
        • Malvehy J.
        • Garcia R.
        A new total body scanning system for automatic change detection in multiple pigmented skin lesions.
        IEEE Trans Med Imaging. 2015; 34: 317-338
        • Levy J.L.
        • Trelles M.A.
        • Levy A.
        • Besson R.
        Photography in dermatology: comparison between slides and digital imaging.
        J Cosmet Dermatol. 2003; 2: 131-134
        • Maguire W.F.
        • Kirkwood J.M.
        Developing agents for the therapeutic prevention of melanoma: Can the assessment of cutaneous precursor lesions help?.
        Future Oncol. Future Medicine. 2020; 16: 413-415
        • Mar V.J.
        • Chamberlain A.J.
        • Kelly J.W.
        • Murray W.K.
        • Thompson J.F.
        Clinical practice guidelines for the diagnosis and management of melanoma: melanomas that lack classical clinical features.
        Medical Journal of Australia. 2017; 207: 348-350
        • Matin R.N.
        • Dinnes J.
        AI-based smartphone apps for risk assessment of skin cancer need more evaluation and better regulation.
        Br J Cancer. 2021; 124: 1749-1750
        • Mcgregor B.
        Automatic registration of images of pigmented skin lesions.
        Pattern Recognition. 1998; 31: 805-817
        • Mirzaalian H.
        • Lee T.K.
        • Hamarneh G.
        Skin lesion tracking using structured graphical models.
        Med Image Anal. 2016; 27: 84-92
        • Morris J.B.
        • Alfonso S.V.
        • Hernandez N.
        • Fernández M.I.
        Use of and intentions to use dermoscopy among physicians in the United States.
        Dermatol Pract Concept. 2017; 7: 7-16
        • Navarro F.
        • Escudero-Viñolo M.
        • Bescós J.
        Accurate Segmentation and Registration of Skin Lesion Images to Evaluate Lesion Change.
        IEEE Journal of Biomedical and Health Informatics. 2019; 23: 501-508
        • Rotemberg V.
        • Kurtansky N.
        • Betz-Stablein B.
        • Caffery L.
        • Chousakos E.
        • Codella N.
        • et al.
        A patient-centric dataset of images and metadata for identifying melanomas using clinical context.
        Sci Data. 2021; 8: 34
        • Schneider S.L.
        • Kohli I.
        • Hamzavi I.H.
        • Council M.L.
        • Rossi A.M.
        • Ozog D.M.
        Emerging imaging technologies in dermatology: Part II: Applications and limitations.
        J Am Acad Dermatol. 2019; 80: 1121-1131
        • Shain A.H.
        • Bastian B.C.
        From melanocytes to melanomas.
        Nat. Rev. Cancer. 2016; 16: 345-358
        • Surjana D.
        • Halliday G.M.
        • Martin A.J.
        • Moloney F.J.
        • Damian D.L.
        Oral nicotinamide reduces actinic keratoses in phase II double-blinded randomized controlled trials.
        J. Invest. Dermatol. 2012; 132: 1497-1500
        • Tahata S.
        • Singh S.V.
        • Lin Y.
        • Hahm E.-R.
        • Beumer J.H.
        • Christner S.M.
        • et al.
        Evaluation of biodistribution of sulforaphane after administration of oral broccoli sprout extract in melanoma patients with multiple atypical nevi.
        Cancer Prev. Res. (Phila.). 2018; 11: 429-438
        • MsPS Tondewad
        • Dale MsMP.
        Remote Sensing Image Registration Methodology: Review and Discussion.
        Procedia Computer Science. 2020; 171: 2390-2399
        • Tschandl P.
        • Codella N.
        • Akay B.N.
        • Argenziano G.
        • Braun R.P.
        • Cabo H.
        • et al.
        Comparison of the accuracy of human readers versus machine-learning algorithms for pigmented skin lesion classification: an open, web-based, international, diagnostic study.
        The Lancet Oncology. 2019; 20: 938-947
        • Vanbelle S.
        A New Interpretation of the Weighted Kappa Coefficients.
        Psychometrika. 2016; 81: 399-410