M³LungXAI-LF: An Explainable Multi-Modal Late Fusion System for Improved Clinical Early Lung Cancer Diagnosis With Automated Medical Reporting

Overview

Lung cancer, the leading cause of cancer-related deaths globally, continues to face limited clinical adoption of machine learning model analysis for diagnosis due to the absence of an integrated diagnosis pipeline and insufficient prediction interpretability. This study focuses on the development of a late-fusion multimodal system integrating multiple data modalities (X-ray, CT scan, and symptoms) to help address clinical challenges and enhance diagnostic accuracy in lung cancer-related applications. The final system employs a weighted decision scheme to combine predictions from each modality, with pre-determined weights based on both empirical performance and clinical relevance. Additionally, a custom lung cancer-oriented clinical note dataset of 10,000 notes was generated using a large language model (LLM) to train the symptom modality natural language processing (NLP) model, with the LLM serving as a strict formatter that adheres to the provided input data to ensure reliability and avoid harmful hallucinations. Throughout the multimodal system, extensive Explainable-AI (XAI) techniques were employed for both global and local explanations, ensuring thorough interpretability and clinical transparency by producing detailed insights into how predictions are made and why specific features are influential. With AUC scores of 97.96% for the X-ray modality, 99.63% for the CT scan modality, and F1-score of 98.10% for the symptom modality, the final multimodal prototype generates automated medical reports that combine heatmaps, annotated images, and clear, evidence-based justifications for its predictions, demonstrating this approach's potential for more reliable and interpretable diagnosis tools in clinical settings.

Competition history

• ISEF 2025 Robotics and Intelligent Machines · Entry ROBO022T

Resources

• ISEF abstract page ↗

Related projects

Closest projects by meaning, across every fair and year in the corpus.