MED-X: An Explainable Multi-Agent Reasoning System for Efficient Diagnostic Decision-Making in Cancerous Polyp Classification Utilizing Multimodal Gastrointestinal Datasets

Overview

Each year, 1.9 million people worldwide are diagnosed with colorectal cancer, and early detection of cancerous polyps is crucial for treatment. Endoscopic imaging is a critical part of identifying polyps to diagnose such cancers early using direct visualization of the gastrointestinal system. Traditional approaches for processing endoscopies are explainable but inefficient, leading to delays in timely treatment. State-of-the-art AI approaches that rely on image classification are hard to explain and focus on a specific modality. There exists a significant gap between AI advances and usage in a clinical setting. We bridge this gap between AI development and real-life diagnostic decision-making by developing MED-X, a multi-agent AI system that analyzes an endoscopic image, pathology reports, and patient-specific data to arrive at diagnostic conclusions. MED-X is both explainable and efficient when making accurate diagnostic decisions given multimodal data. Different multimodal vision model-based agents were trained and fine-tuned on gastrointestinal data. Explainability arises out of our use of knowledge graphs, reasoning models and multi-agent decision summarization. We built an agentic system where models collaborate and refine their responses to arrive at a conclusion similar to a team of human experts, demonstrating a 285% better accuracy on diagnosis of the Kudo and Paris classifications than state-of-the-art multimodal models. We compare MED-X's performance against GI specialists, achieving 2x better accuracy compared to the doctors. This multi-agent system is the first of its kind, where an agentic framework simulates the work of doctors in a diagnostic setting. By surpassing the capabilities of GI specialists, MED-X presents a groundbreaking diagnostic assistant.

Competition history

• ISEF 2025 Translational Medical Science · Entry TMED052

Resources

• ISEF abstract page ↗

Related projects

Closest projects by meaning, across every fair and year in the corpus.