A Holistic Multi-Modal GenAI System: Early Detection and Predictive Treatment Monitoring Using Chest X-rays and Cough Audio for Tuberculosis
ISEF · 2024 Biomedical Engineering
Overview
Tuberculosis (TB), a bacterial infectious disease, is one of the top 10 causes of mortality worldwide in low-income countries, resulting in approximately 10 million new infections and 1.4 million deaths. I updated the ML model to include multi-modality (text and audio input) as well as additional feature extraction (mel-spectrograms) and data augmentation (IR-convolution). I created a mobile and web app to integrate the model, and the results are available within 15 seconds. After preliminary testing, the model achieved an area under the receiving operator characteristic curve (AUROC) of 88%, surpassing the WHO requirements for screening tests. I also integrated my new research on chest radiography (CXR) tools. I created a novel 2D convolutional neural network (2D-CNN) to identify and forecast subsequent incident TB using CXR. Predicting the risk of active TB long before symptoms enables preventive treatment that can be administered earlier (6-12 months before symptoms). I trained the model on the O2 high-performance cluster. I implemented Focal Loss and class weightage to counteract the high-class imbalance. The optimal 2D-CNN included a Gated Activation Unit and a Multi-Head Self-Attention (heads=8), performing with a specificity of 93.7%, a sensitivity of 75.8%, and an AUC ROC of 83.7%, showcasing the strong potential for using CXRs in contact tracing. I have integrated these two models into a generative AI system powered by a custom LLM I created using the LLaMa2 open-source and a Retrieval Augumentaed Generation vector database. This provides more up-to-date knowledge and specific facts for the LLM, creating a comprehensive patient treatment and monitoring system, and increasing access to quality healthcare worldwide, especially in low/middle-income areas.
Competition history
- ISEF 2024
Resources
Related projects
ISEF · 2023
Improving Early Diagnosis and Treatment Monitoring of Tuberculosis With Novel Machine Learning Cough Analysis
ISEF · 2021
A Low-Cost Computer-Aided Lung Auscultation Apparatus and Automated Diagnosis of Respiratory Illnesses
CWSF · 2026
An Efficient AI-Driven Tool for Pediatric Pneumonia Detection
CWSF · 2026
Operation Oracle: Multi-Modal, Explainable AI Systems for Accessible Cancer Detection
Closest projects by meaning, across every fair and year in the corpus.
Source: Regeneron International Science and Engineering Fair