OcuScan: A Novel Deep Learning-Based Ocular Disease Detection System

My name is Gaurav Chopra. I’m in grade 10, and my project focussed on the development of a novel deep learning-based tool that can differentiate normal retinal images from abnormal ones. Diagnosing ocular diseases quickly and accurately is crucial because prolonged disease can lead to permanent eye damage. However, this isn’t always feasible in areas with limited ophthalmologists. While deep learning has been used to automate this process, most pre-existing systems are rendered clinically inapplicable because they are limited to training-specific pathologies. My work overcomes this challenge by improving the generalizability and robustness of pre-existing deep learning networks. My model demonstrates high generalizability, suggesting significantly improved efficacy in healthcare. Future work will include introducing pathology-specific detection to increase the automation capabilities of this detection system.

Introduction

It is estimated at least 2.2 billion people suffer from visual impairments worldwide [1]. Diagnosing ocular diseases quickly and accurately is essential, as prolonged disease will lead to permanent eye damage [2]. Many patient populations, however, suffer from increasingly long diagnostic wait times of 2-3 months [3], making these patients extremely susceptible to permanent eye damage.

Deep learning has been used to automate ocular diagnosis using fundus (retinal) images to accelerate wait times [4]. While these systems demonstrate very high efficacy on the specific subset of diseases present during training [5], systemic evaluations have revealed these models are unable detect pathologies outside of their training scope, often classifying external diseases as normal [6]. This is problematic because it is computationally infeasible to train deep learning systems on every existing ocular pathology, and false normal results are particularly detrimental. Consequently, further ophthalmologist-based diagnosis is required, negating the effect of these models. One study revealed the less than 5% adoption rate of an FDA-approved diabetic retinopathy detection system [7].

Solution

This work presents OcuScan, a novel detection system that differentiates abnormal from normal fundus images. Unlike pre-existing systems, OcuScan can detect the presence of pathologies within and outside of its training scope. OcuScan employs a novel generalizable approach to facilitate pattern recognition and comparative analysis instead of traditional feature recognition, while still utilizing convolutional operations for image analysis through automatic feature extraction. This approach reduces OcuScan's reliance on pathology-specific image features, thereby improving its ability to generalize on external pathologies.

Data Collection

I created four datasets from the Eye Disease Diagnosis and Fundus Synthesis (EDDFS) database [8-10] and the Eye Disease Image Dataset (EDID) [11]. Three of the datasets were used for training with an 80%/20% train/validation split, while EDID test was used exclusively for testing (fig. 1-3).

Model Construction

OcuScan performs fundus classification using a novel convolutional neural network (fig. 5). During training, new input images are synthesized by generating unique, random variations of the original input images based on pre-defined augmentation techniques. From here, feature extraction is conducted by passing input images through two convolutional layer-based heads simultaneously, each outputting multi-dimensional maps of relevant features defined during training. Within each head, an attention mechanism was inserted (fig. 4), facilitating channel (red-green-blue values) attention and spatial attention to improve OcuScan's ability to detect regions of interest. Both feature maps are flattened into a one-dimensional feature vector before being fused together. The concatenated vector passes through 2 fully-connected layers before being passed into the output layer. OcuScan outputs a probability score between 0 (abnormal) and 1 (normal) using the sigmoid activation function. This value is compared to a pre-defined threshold (0.451) to make a classification decision.

Training

Training was split into six phases totaling to 158 epochs (rounds of training). During each epoch, batches of 32 images are propagated through the network, which outputs a probability score for each image. A loss is quantified using the binary cross-entropy function between the predicted distribution and the true binary distribution. The function is differentiated with respect to each parameter of the network. A mathematically optimized learning rate is applied to each derivative to determine the value by which to adjust each parameter to minimize loss. This process is repeated for each batch of images in the training dataset.

Generalization Capacity

To evaluate OcuScan’s capacity for detecting the presence of pathologies outside of its training scope, its generalization capacity was assessed on the test dataset. Images were stratified by pathology. The model was evaluated on each pathologic group of images to produce an overall pathologic performance distribution based on accuracy (fig. 6), which was then assessed by variance. Lower variance is indicative of higher generalization. To facilitate a multi-faceted evaluation, symmetry, statistical anomalies, and the coefficient of variation were calculated and interpreted. Outliers exist in any dataset as: Q1 – (1.5 x IQR) or Q3 + (1.5 x IQR). The coefficient of variation (CV) is a unitless measure that calculates the ratio of the standard deviation (SD) to the mean (SD/mean), representing the relative dispersion of data points relative to their average.

Mean: 81.9%

Median: 80.0%

SD: 14.4%

CV: 17.6%

Statistical Anomalies: none (0)

The mean and median were highly similar (+/- 1.9%), indicating symmetry without skewedness. The CV was 17.6%, which falls well below the 100% threshold for distributions with high variance.

OcuScan Performance

OcuScan's binary fundus classification performance on the test dataset was evaluated using the following metrics, statistically computed from the confusion matrix in Fig. 7:

Accuracy: 85.5%

Precision: 83.1%

Recall: 89.1%

F1-Score: 86.0%

Sensitivity: 89.1%

Specificity: 81.9%

Ranking quality was evaluated from the quantification of two threshold-independent curves. The Receiver Operating Characteristic (ROC) curve (fig. 8) was generated by mapping sensitivity (true positive rate) against the false positive rate (1 - specificity) as the threshold decreased from 1.0 to 0.0. The area under the curve (AUC) was calculated as 92.6%, indicating OcuScan ranked 92.6% of all normal images above all abnormal images. The Precision-Recall (PR) curve (fig. 9) was generated by mapping precision against recall as the threshold decreased from 1.0 to 0.0. OcuScan achieved a 92.0% AUC, indicating how well it can identify normal images while minimizing false normals.

OcuScan's proportion 95% confidence interval was calculated:

p̂ ± z(√ p̂(1 - p̂) / n), where:

p̂ is the sample proportion (0.855), z is the critical value based on the confidence level (1.96), and n is the sample size (1026).

With 95% confidence, OcuScan's true accuracy is between 83.3% and 87.7%.

Explainability

OcuScan's explainability was qualitatively evaluated using saliency maps for each diagnostic group of images in the test dataset. Gradient-weighted class activation mapping was used to create a heatmap, highlighting the specific regions of each image that most influenced OcuScan's final prediction. It can be observed that for most diagnoses, OcuScan correctly focuses its attention on the areas of the fundus where an abnormality is present (fig. 10). Each feature extraction head appears to focus on different regions of the image, demonstrating high generalizability. OcuScan's learned features are distributed across most regions of the image instead of being concentrated on certain areas.

This work presented OcuScan, a novel ocular disease screening tool. OcuScan is powered by a convolutional neural network (CNN) architecture that employs a novel approach to improve generalization compared to traditional CNN models. Pre-existing systems are fundamentally limited to their training data, classifying external pathologies as "normal". However, OcuScan is the first detection system that can accurately detect the presence of pathologies outside of its training scope.

OcuScan was evaluated on its generalization capacity, performance on the test data, and its explainability. Statistical analysis on OcuScan's pathologic performance distribution was conducted to assess the model's generalization capacity. The distribution showed very little variance, with a key result being the coefficient of variation of 17.6% falling far below the threshold of 100%. This indicates exceptional generalization, and thus, high performance on external pathologies. Derived from the results, we see that OcuScan performed with 83.1% positive predictive value, and 88.3% negative predictive value (NPV). Additionally, OcuScan achieved 89.1% sensitivity, and 81.9% specificity. This means that OcuScan has an 89.1% chance of correctly classifying a normal fundus, and an 81.9% chance of correctly classifying an abnormal fundus. Finally, OcuScan's explainability was evaluated using gradient-weighted class activation mapping. Qualitative results indicate OcuScan correctly places its attention for most diagnoses. This means that OcuScan is accurately identifying the presence of pathologies within the eye fundus for the correct reasons, helping to mitigate the “black box” problem, where deep learning models cannot explain their reasoning when performing predictions.

Future work includes evaluating OcuScan on more diagnoses to better assess its generalizability. There were a limited number of diagnoses within eye fundus data publicly available. Additionally, despite employing gradient-weighted class activation maps, the interpretability of OcuScan's results remains ambiguous within a clinical setting. To fully mitigate this problem, a combination of natural language processing and computer vision will be required. Furthermore, comparing OcuScan to publicly available CNN benchmarks should be conducted to evaluate OcuScan's robustness. Finally, OcuScan should be leveraged within a user-friendly, Internet-accessible application for ease of use and universal accessibility.

I would like to acknowledge Nelson Burton for providing invaluable guidance and support at various points during this research, particularly during the development and training of OcuScan.

WHO. (n.d.). Vision Impairment and blindness. World Health Organization. https://www.who.int/news-room/fact-sheets/detail/blindness-and-visual-impairment

National Academies of Sciences, Engineering, and Medicine; Health and Medicine Division; Board on Population Health and Public Health Practice; Committee on Public Health Approaches to Reduce Vision Impairment and Promote Eye Health; Welp A, Woodbury RB, McCoy MA, et al., editors. Making Eye Health a Population Health Imperative: Vision for Tomorrow. Washington (DC): National Academies Press (US); 2016 Sep 15. 2, Understanding the Epidemiology of Vision Loss and Impairment in the United States. Available from: https://www.ncbi.nlm.nih.gov/books/NBK402366/

Thind, A., Stewart, M., Manuel, D., Freeman, T., Terry, A., Chevendra, V., Maddocks, H., & Marshall, N. (2012, August 8). What are wait times to see a specialist? an analysis of 26,942 referrals in Southwestern Ontario. Healthcare policy = Politiques de sante. https://pmc.ncbi.nlm.nih.gov/articles/PMC3430156/#:~:text=(Statacorp%202010).-,Results,median%20wait%20=%20103%20day

Hussein SA, Farouk AA, Saeid MM. Intelligent retinal disease detection using deep learning. Sci Rep. 2025 Dec 8;15(1):43282. doi: 10.1038/s41598-025-28376-w. PMID: 41354757; PMCID: PMC12686508.

Xie, H., Li, Z., Wu, C., Zhao, Y., Lin, C., Wang, Z., Wang, C., Gu, Q., Wang, M., Zheng, Q., Jiang, J., & Chen, W. (2023, July 28). Deep learning for detecting visually impaired cataracts using fundus images. Frontiers in cell and developmental biology. https://pmc.ncbi.nlm.nih.gov/articles/PMC10416247/#s1

Shi, S., Malhi, I., Tran, K., Ng, A.Y. &amp; Rajpurkar, P. (2021). Unseen Disease Detection for Deep Learning Interpretation of Chest X-rays. <i>Proceedings of the Fourth Conference on Medical Imaging with Deep Learning</i>, in <i>Proceedings of Machine Learning Research</i> 143:699-712 Available from https://proceedings.mlr.press/v143/shi21a.html

Shah, S. A., Sokol, J. T., Wai, K. M., et al. (2024). Use of artificial intelligence–based detection of diabetic retinopathy in the US. JAMA Ophthalmology. Advance online publication. https://doi.org/10.1001/jamaophthalmol.2024.4493

Wang, T., Liao, G., Chen, L., Zhuang, Y., Zhou, S., Yuan, Q., Han, L., Wu, S., Chen, K., Wang, B., Mi, J., Gao, Y., Lin, J., & Zhang, M. (2023, April). Intelligent diagnosis of multiple peripheral retinal lesions in ultra-widefield fundus images based on Deep Learning. Ophthalmology and therapy. https://pmc.ncbi.nlm.nih.gov/articles/PMC9872743/

Xia, X., Zhan, K., Li, Y., Xiao, G., Yan, J., Huang, Z., Huang, G., & Fang, Y. (2022). Eye disease diagnosis and fundus synthesis: A large-scale dataset and benchmark. In 2022 IEEE 24th International Workshop on Multimedia Signal Processing (MMSP) (pp. 1–6). IEEE. https://doi.org/10.1109/MMSP55362.2022.9949547

Xia, X., Li, Y., Xiao, G., Zhan, K., Yan, J., Cai, C., Fang, Y., & Huang, G. (2024). Benchmarking deep models on retinal fundus disease diagnosis and a large-scale dataset. Signal Processing: Image Communication, 127, 117151. https://doi.org/10.1016/j.image.2024.117151

Riadur Rashid, Mohammad ; Sharmin, Shayla ; Khatun, Tania; Hasan, Md Zahid; Shorif Uddin , Mohammad (2024), “Eye Disease Image Dataset”, Mendeley Data, V1, doi: 10.17632/s9bfhswzjb.1

Woo, S., Park, J., Lee, J.-Y., & Kweon, I. S. (2018, July 18). CBAM: Convolutional Block Attention Module. arXiv.org. https://arxiv.org/abs/1807.06521