Mixed Reality Application for Pancreatic Surgery

Overview

In 2024 the American Cancer Society (ACS) reported a 78% death rate for pancreatic cancer. The 5-year survival rate for pancreatic cancer is 13%. ACS states the only effective method known to “cure” pancreatic cancer is surgery. Pancreatic cancer surgery has approximately a 40% complication rate (Ho, Choon-Kiat, NIH). The goal of this research is to help with surgical pre-planning/operation. To achieve this, U-Nets were trained to segment pancreas, liver, hepatic vessels, and their respective tumors from CT scans as well as integrated into a browser application. A VR headset housed a mixed reality (MR) application that notifies surgeons of 9 vital organs near the pancreas using a YOLO. Three U-Net models were trained for each task (9 models) for each segmentation task and three YOLOs were trained. For the pancreas segmentation task, the nnU-Net had the highest validation dice coefficient of 0.9576. For the hepatic vessel task, the nnU-Net had the highest coefficient of 0.9641. For the liver task, the 2D Attention U-Net had the highest coefficient of 0.9823. For the organ detection task, the YOLOv8 model had the highest mean average precision (mAP) of 89%. 1,200 trials were conducted afterwards to validate all the models' performance on unseen images. The three best performing U-Nets were successfully converted into a browser application where CT scans can be uploaded for the models to segment their respective organs and tumors. A MR application was successfully developed using the YOLOv8 that can detect nine major organs involved in pancreatic surgery in real time. The developed application was reviewed by local surgeons to see how useful the application was and surgeons reported that it could reduce the complication rate by 10 to 30%.

Competition history

• ISEF 2025 Translational Medical Science · Entry TMED050

Resources

• ISEF abstract page ↗

Related projects

Closest projects by meaning, across every fair and year in the corpus.