Skin cancer classification via convolutional neural networks: systematic review of studies involving human experts

Sarah Haggenmüller; Roman C. Maron; Achim Hekler; Jochen S. Utikal; Catarina Barata; Raymond L. Barnhill; Helmut Beltraminelli; Carola Berking; Brigid Betz-Stablein; Andreas Blum; Stephan A. Braun; Richard Carr; Marc Combalia; Maria Teresa Fernandez-Figueras; Gerardo Ferrara; Sylvie Fraitag; Lars E. French; Frank F. Gellrich; Kamran Ghoreschi; Matthias Goebeler; Pascale Guitera; Holger A. Haenssle; Sebastian Haferkamp; Lucie Heinzerling; Markus V. Heppt; Franz J. Hilke; Sarah Hobelsberger; Dieter Krahl; Heinz Kutzner; Aimilios Lallas; Konstantinos Liopyris; Mar Llamas-Velasco; Josep Malvehy; Friedegund Meier; Cornelia S.L. Müller; Alexander A. Navarini; Cristián Navarrete-Dechent; Antonio Perasole; Gabriela Poch; Sebastian Podlipnik; Luis Requena; Veronica M. Rotemberg; Andrea Saggini; Omar P. Sangueza; Carlos Santonja; Dirk Schadendorf; Bastian Schilling; Max Schlaak; Justin G. Schlager; Mildred Sergon; Wiebke Sondermann; H. Peter Soyer; Hans Starz; Wilhelm Stolz; Esmeralda Vale; Wolfgang Weyers; Alexander Zink; Eva Krieghoff-Henning; Jakob N. Kather; Christof von Kalle; Daniel B. Lipka; Stefan Fröhling; Axel Hauschild; Harald Kittler; Titus J. Brinker

doi:10.1016/j.ejca.2021.06.049

Skin cancer classification via convolutional neural networks: systematic review of studies involving human experts

Sarah Haggenmüller, Roman C. Maron, Achim Hekler, Jochen S. Utikal, Catarina Barata, Raymond L. Barnhill, Helmut Beltraminelli, Carola Berking, Brigid Betz-Stablein, Andreas Blum, Stephan A. Braun, Richard Carr, Marc Combalia, Maria Teresa Fernandez-Figueras, Gerardo Ferrara, Sylvie Fraitag, Lars E. French, Frank F. Gellrich, Kamran Ghoreschi, Matthias GoebelerPascale Guitera, Holger A. Haenssle, Sebastian Haferkamp, Lucie Heinzerling, Markus V. Heppt, Franz J. Hilke, Sarah Hobelsberger, Dieter Krahl, Heinz Kutzner, Aimilios Lallas, Konstantinos Liopyris, Mar Llamas-Velasco, Josep Malvehy, Friedegund Meier, Cornelia S.L. Müller, Alexander A. Navarini, Cristián Navarrete-Dechent, Antonio Perasole, Gabriela Poch, Sebastian Podlipnik, Luis Requena, Veronica M. Rotemberg, Andrea Saggini, Omar P. Sangueza, Carlos Santonja, Dirk Schadendorf, Bastian Schilling, Max Schlaak, Justin G. Schlager, Mildred Sergon, Wiebke Sondermann, H. Peter Soyer, Hans Starz, Wilhelm Stolz, Esmeralda Vale, Wolfgang Weyers, Alexander Zink, Eva Krieghoff-Henning, Jakob N. Kather, Christof von Kalle, Daniel B. Lipka, Stefan Fröhling, Axel Hauschild, Harald Kittler, Titus J. Brinker^*

^*Corresponding author for this work

Research output: Contribution to journal › Article › Research › peer-review

191 Citations (Scopus)

Abstract

Background: Multiple studies have compared the performance of artificial intelligence (AI)–based models for automated skin cancer classification to human experts, thus setting the cornerstone for a successful translation of AI-based tools into clinicopathological practice. Objective: The objective of the study was to systematically analyse the current state of research on reader studies involving melanoma and to assess their potential clinical relevance by evaluating three main aspects: test set characteristics (holdout/out-of-distribution data set, composition), test setting (experimental/clinical, inclusion of metadata) and representativeness of participating clinicians. Methods: PubMed, Medline and ScienceDirect were screened for peer-reviewed studies published between 2017 and 2021 and dealing with AI-based skin cancer classification involving melanoma. The search terms skin cancer classification, deep learning, convolutional neural network (CNN), melanoma (detection), digital biomarkers, histopathology and whole slide imaging were combined. Based on the search results, only studies that considered direct comparison of AI results with clinicians and had a diagnostic classification as their main objective were included. Results: A total of 19 reader studies fulfilled the inclusion criteria. Of these, 11 CNN-based approaches addressed the classification of dermoscopic images; 6 concentrated on the classification of clinical images, whereas 2 dermatopathological studies utilised digitised histopathological whole slide images. Conclusions: All 19 included studies demonstrated superior or at least equivalent performance of CNN-based classifiers compared with clinicians. However, almost all studies were conducted in highly artificial settings based exclusively on single images of the suspicious lesions. Moreover, test sets mainly consisted of holdout images and did not represent the full range of patient populations and melanoma subtypes encountered in clinical practice.

Original language	English
Pages (from-to)	202-216
Number of pages	15
Journal	European Journal of Cancer
Volume	156
DOIs	https://doi.org/10.1016/j.ejca.2021.06.049
Publication status	Published - Oct 2021
Externally published	Yes

Keywords

Artificial intelligence
Convolutional neural network(s)
Deep learning
Dermatology
Digital biomarkers
Machine learning
Malignant melanoma
Skin cancer classification

Access to Document

10.1016/j.ejca.2021.06.049

Cite this

Haggenmüller, S., Maron, R. C., Hekler, A., Utikal, J. S., Barata, C., Barnhill, R. L., Beltraminelli, H., Berking, C., Betz-Stablein, B., Blum, A., Braun, S. A., Carr, R., Combalia, M., Fernandez-Figueras, M. T., Ferrara, G., Fraitag, S., French, L. E., Gellrich, F. F., Ghoreschi, K., ... Brinker, T. J. (2021). Skin cancer classification via convolutional neural networks: systematic review of studies involving human experts. European Journal of Cancer, 156, 202-216. https://doi.org/10.1016/j.ejca.2021.06.049

@article{37d6a724c29d41df8653ad102d13fa7c,

title = "Skin cancer classification via convolutional neural networks: systematic review of studies involving human experts",

abstract = "Background: Multiple studies have compared the performance of artificial intelligence (AI)–based models for automated skin cancer classification to human experts, thus setting the cornerstone for a successful translation of AI-based tools into clinicopathological practice. Objective: The objective of the study was to systematically analyse the current state of research on reader studies involving melanoma and to assess their potential clinical relevance by evaluating three main aspects: test set characteristics (holdout/out-of-distribution data set, composition), test setting (experimental/clinical, inclusion of metadata) and representativeness of participating clinicians. Methods: PubMed, Medline and ScienceDirect were screened for peer-reviewed studies published between 2017 and 2021 and dealing with AI-based skin cancer classification involving melanoma. The search terms skin cancer classification, deep learning, convolutional neural network (CNN), melanoma (detection), digital biomarkers, histopathology and whole slide imaging were combined. Based on the search results, only studies that considered direct comparison of AI results with clinicians and had a diagnostic classification as their main objective were included. Results: A total of 19 reader studies fulfilled the inclusion criteria. Of these, 11 CNN-based approaches addressed the classification of dermoscopic images; 6 concentrated on the classification of clinical images, whereas 2 dermatopathological studies utilised digitised histopathological whole slide images. Conclusions: All 19 included studies demonstrated superior or at least equivalent performance of CNN-based classifiers compared with clinicians. However, almost all studies were conducted in highly artificial settings based exclusively on single images of the suspicious lesions. Moreover, test sets mainly consisted of holdout images and did not represent the full range of patient populations and melanoma subtypes encountered in clinical practice.",

keywords = "Artificial intelligence, Convolutional neural network(s), Deep learning, Dermatology, Digital biomarkers, Machine learning, Malignant melanoma, Skin cancer classification",

author = "Sarah Haggenm{\"u}ller and Maron, {Roman C.} and Achim Hekler and Utikal, {Jochen S.} and Catarina Barata and Barnhill, {Raymond L.} and Helmut Beltraminelli and Carola Berking and Brigid Betz-Stablein and Andreas Blum and Braun, {Stephan A.} and Richard Carr and Marc Combalia and Fernandez-Figueras, {Maria Teresa} and Gerardo Ferrara and Sylvie Fraitag and French, {Lars E.} and Gellrich, {Frank F.} and Kamran Ghoreschi and Matthias Goebeler and Pascale Guitera and Haenssle, {Holger A.} and Sebastian Haferkamp and Lucie Heinzerling and Heppt, {Markus V.} and Hilke, {Franz J.} and Sarah Hobelsberger and Dieter Krahl and Heinz Kutzner and Aimilios Lallas and Konstantinos Liopyris and Mar Llamas-Velasco and Josep Malvehy and Friedegund Meier and M{\"u}ller, {Cornelia S.L.} and Navarini, {Alexander A.} and Cristi{\'a}n Navarrete-Dechent and Antonio Perasole and Gabriela Poch and Sebastian Podlipnik and Luis Requena and Rotemberg, {Veronica M.} and Andrea Saggini and Sangueza, {Omar P.} and Carlos Santonja and Dirk Schadendorf and Bastian Schilling and Max Schlaak and Schlager, {Justin G.} and Mildred Sergon and Wiebke Sondermann and Soyer, {H. Peter} and Hans Starz and Wilhelm Stolz and Esmeralda Vale and Wolfgang Weyers and Alexander Zink and Eva Krieghoff-Henning and Kather, {Jakob N.} and {von Kalle}, Christof and Lipka, {Daniel B.} and Stefan Fr{\"o}hling and Axel Hauschild and Harald Kittler and Brinker, {Titus J.}",

note = "Publisher Copyright: {\textcopyright} 2021 The Author(s)",

year = "2021",

month = oct,

doi = "10.1016/j.ejca.2021.06.049",

language = "English",

volume = "156",

pages = "202--216",

journal = "European Journal of Cancer",

issn = "0959-8049",

publisher = "Elsevier Ltd.",

}

Haggenmüller, S, Maron, RC, Hekler, A, Utikal, JS, Barata, C, Barnhill, RL, Beltraminelli, H, Berking, C, Betz-Stablein, B, Blum, A, Braun, SA, Carr, R, Combalia, M, Fernandez-Figueras, MT, Ferrara, G, Fraitag, S, French, LE, Gellrich, FF, Ghoreschi, K, Goebeler, M, Guitera, P, Haenssle, HA, Haferkamp, S, Heinzerling, L, Heppt, MV, Hilke, FJ, Hobelsberger, S, Krahl, D, Kutzner, H, Lallas, A, Liopyris, K, Llamas-Velasco, M, Malvehy, J, Meier, F, Müller, CSL, Navarini, AA, Navarrete-Dechent, C, Perasole, A, Poch, G, Podlipnik, S, Requena, L, Rotemberg, VM, Saggini, A, Sangueza, OP, Santonja, C, Schadendorf, D, Schilling, B, Schlaak, M, Schlager, JG, Sergon, M, Sondermann, W, Soyer, HP, Starz, H, Stolz, W, Vale, E, Weyers, W, Zink, A, Krieghoff-Henning, E, Kather, JN, von Kalle, C, Lipka, DB, Fröhling, S, Hauschild, A, Kittler, H & Brinker, TJ 2021, 'Skin cancer classification via convolutional neural networks: systematic review of studies involving human experts', European Journal of Cancer, vol. 156, pp. 202-216. https://doi.org/10.1016/j.ejca.2021.06.049

TY - JOUR

T1 - Skin cancer classification via convolutional neural networks

T2 - systematic review of studies involving human experts

AU - Haggenmüller, Sarah

AU - Maron, Roman C.

AU - Hekler, Achim

AU - Utikal, Jochen S.

AU - Barata, Catarina

AU - Barnhill, Raymond L.

AU - Beltraminelli, Helmut

AU - Berking, Carola

AU - Betz-Stablein, Brigid

AU - Blum, Andreas

AU - Braun, Stephan A.

AU - Carr, Richard

AU - Combalia, Marc

AU - Fernandez-Figueras, Maria Teresa

AU - Ferrara, Gerardo

AU - Fraitag, Sylvie

AU - French, Lars E.

AU - Gellrich, Frank F.

AU - Ghoreschi, Kamran

AU - Goebeler, Matthias

AU - Guitera, Pascale

AU - Haenssle, Holger A.

AU - Haferkamp, Sebastian

AU - Heinzerling, Lucie

AU - Heppt, Markus V.

AU - Hilke, Franz J.

AU - Hobelsberger, Sarah

AU - Krahl, Dieter

AU - Kutzner, Heinz

AU - Lallas, Aimilios

AU - Liopyris, Konstantinos

AU - Llamas-Velasco, Mar

AU - Malvehy, Josep

AU - Meier, Friedegund

AU - Müller, Cornelia S.L.

AU - Navarini, Alexander A.

AU - Navarrete-Dechent, Cristián

AU - Perasole, Antonio

AU - Poch, Gabriela

AU - Podlipnik, Sebastian

AU - Requena, Luis

AU - Rotemberg, Veronica M.

AU - Saggini, Andrea

AU - Sangueza, Omar P.

AU - Santonja, Carlos

AU - Schadendorf, Dirk

AU - Schilling, Bastian

AU - Schlaak, Max

AU - Schlager, Justin G.

AU - Sergon, Mildred

AU - Sondermann, Wiebke

AU - Soyer, H. Peter

AU - Starz, Hans

AU - Stolz, Wilhelm

AU - Vale, Esmeralda

AU - Weyers, Wolfgang

AU - Zink, Alexander

AU - Krieghoff-Henning, Eva

AU - Kather, Jakob N.

AU - von Kalle, Christof

AU - Lipka, Daniel B.

AU - Fröhling, Stefan

AU - Hauschild, Axel

AU - Kittler, Harald

AU - Brinker, Titus J.

PY - 2021/10

Y1 - 2021/10

N2 - Background: Multiple studies have compared the performance of artificial intelligence (AI)–based models for automated skin cancer classification to human experts, thus setting the cornerstone for a successful translation of AI-based tools into clinicopathological practice. Objective: The objective of the study was to systematically analyse the current state of research on reader studies involving melanoma and to assess their potential clinical relevance by evaluating three main aspects: test set characteristics (holdout/out-of-distribution data set, composition), test setting (experimental/clinical, inclusion of metadata) and representativeness of participating clinicians. Methods: PubMed, Medline and ScienceDirect were screened for peer-reviewed studies published between 2017 and 2021 and dealing with AI-based skin cancer classification involving melanoma. The search terms skin cancer classification, deep learning, convolutional neural network (CNN), melanoma (detection), digital biomarkers, histopathology and whole slide imaging were combined. Based on the search results, only studies that considered direct comparison of AI results with clinicians and had a diagnostic classification as their main objective were included. Results: A total of 19 reader studies fulfilled the inclusion criteria. Of these, 11 CNN-based approaches addressed the classification of dermoscopic images; 6 concentrated on the classification of clinical images, whereas 2 dermatopathological studies utilised digitised histopathological whole slide images. Conclusions: All 19 included studies demonstrated superior or at least equivalent performance of CNN-based classifiers compared with clinicians. However, almost all studies were conducted in highly artificial settings based exclusively on single images of the suspicious lesions. Moreover, test sets mainly consisted of holdout images and did not represent the full range of patient populations and melanoma subtypes encountered in clinical practice.

AB - Background: Multiple studies have compared the performance of artificial intelligence (AI)–based models for automated skin cancer classification to human experts, thus setting the cornerstone for a successful translation of AI-based tools into clinicopathological practice. Objective: The objective of the study was to systematically analyse the current state of research on reader studies involving melanoma and to assess their potential clinical relevance by evaluating three main aspects: test set characteristics (holdout/out-of-distribution data set, composition), test setting (experimental/clinical, inclusion of metadata) and representativeness of participating clinicians. Methods: PubMed, Medline and ScienceDirect were screened for peer-reviewed studies published between 2017 and 2021 and dealing with AI-based skin cancer classification involving melanoma. The search terms skin cancer classification, deep learning, convolutional neural network (CNN), melanoma (detection), digital biomarkers, histopathology and whole slide imaging were combined. Based on the search results, only studies that considered direct comparison of AI results with clinicians and had a diagnostic classification as their main objective were included. Results: A total of 19 reader studies fulfilled the inclusion criteria. Of these, 11 CNN-based approaches addressed the classification of dermoscopic images; 6 concentrated on the classification of clinical images, whereas 2 dermatopathological studies utilised digitised histopathological whole slide images. Conclusions: All 19 included studies demonstrated superior or at least equivalent performance of CNN-based classifiers compared with clinicians. However, almost all studies were conducted in highly artificial settings based exclusively on single images of the suspicious lesions. Moreover, test sets mainly consisted of holdout images and did not represent the full range of patient populations and melanoma subtypes encountered in clinical practice.

KW - Artificial intelligence

KW - Convolutional neural network(s)

KW - Deep learning

KW - Dermatology

KW - Digital biomarkers

KW - Machine learning

KW - Malignant melanoma

KW - Skin cancer classification

UR - http://www.scopus.com/inward/record.url?scp=85110598997&partnerID=8YFLogxK

U2 - 10.1016/j.ejca.2021.06.049

DO - 10.1016/j.ejca.2021.06.049

M3 - Article

C2 - 34509059

AN - SCOPUS:85110598997

SN - 0959-8049

VL - 156

SP - 202

EP - 216

JO - European Journal of Cancer

JF - European Journal of Cancer

ER -

Skin cancer classification via convolutional neural networks: systematic review of studies involving human experts

Abstract

Keywords

Access to Document

Other files and links

Fingerprint

Cite this