WearWise: Empowering Visually Impaired Individuals in Clothing Management through an Integrated System of Deep Learning and Large Multimodal Model

Overview

Artificial intelligence has developed quickly and improved daily life for the majority of individuals, yet accessibility challenges persist for the visually impaired. Existing assistive technology faces high costs, inaccessible interfaces, and fails to adequately address key needs such as stain detection. WearWise, a mobile application that delivers integrated clothing assistance through a combined approach of a deep learning model and a Large Multimodal API, can detect clothing in real-time, provide audio, natural-language descriptions of clothing, and identify stains. The deep learning model for clothing detection was built on the YOLOv11 framework from a labeled dataset across 11 types of clothing. When clothing is detected, it automatically captures pictures and verifies the output using a Large Multimodal Model (LMM) API. The Stain detection system utilizes a stain detection deep learning model developed using 3 datasets for solid-colored, patterned, and stained clothing items. The app features a voice-only interface, utilizing text-to-speech and speech-to-text to guide the user. The testing results demonstrated that the clothing detection deep learning model achieved an 85.82% F1 score, and the overall system, integrating Gemini-2.0-Flash, achieved 100.0% accuracy from in-app testing. The stain detection model system achieved a 92% F1 score in detecting stains. This portable, cost-free solution revolutionizes clothing assistance for visually impaired users, increasing independence and accessibility.

Competition history

• ISEF 2025 Systems Software · Entry SOFT036

Resources

• ISEF abstract page ↗

Related projects

Closest projects by meaning, across every fair and year in the corpus.