Deep Learning Multi Modal Melanoma Detection

Overview

The visual similarity of melanoma and seborrheic keratosis has made it difficult for elderly patients with disabilities to know when to seek medical attention, contributing to the metastasis of melanoma. In this project, we use multimodal deep learning techniques to distinguish between melanoma and seborrheic keratosis. Background research and past studies to diagnose skin disorders like acne versus eczema led to the hypothesis that the ResNet50 model combined with structured metadata would provide the best accuracy in distinguishing between the two skin conditions due to its ability to perform accurate texture analysis such as border patterns. The strategy is three-fold: (1) utilize patient image data to train and test three deep learning models using transfer learning (ResNet50, InceptionV3, and VGG16) and one author constructed model, (2) use patient metadata to train and test a deep learning metadata model, and (3) use non linear least squares analysis to specify optimal weights for combining predictions from the best performing image model with those from the metadata model. The accuracy of the combined model was 87% on test data with no lesion ID overlap between test and train. Results from this experiment could be used to eliminate late diagnosis of melanoma via easy, ubiquitous access to a smartphone app. Seeking early attention is vital to prevent metastasis. Future work can extend multimodal inputs to include text data in addition to the existing inputs (subjective data pertaining to how the patient felt over a certain period of time) thereby allowing the deep learning model to learn from real hospital settings.

Competition history

• ISEF 2024 Translational Medical Science · Entry TMED061 · Los Angeles, California, United States

Resources

• ISEF abstract page ↗

Related projects

Closest projects by meaning, across every fair and year in the corpus.