vAssist: A Device for the Visually Impaired to Improve In-Store Shopping Using Computer Vision and AI-enabled Routing

Overview

Objectives Grocery shopping is a regular routine in our daily lives. However, the experience can be exceptionally challenging for individuals confronting vision impairment or blindness. The challenges can constitute finding a sighted person to help them maneuver the aisles, avoiding obstacles, identifying items, and scanning barcodes. The COVID-19 pandemic has rendered this task more complicated due to social distancing regulations. My project aspires to combat these struggles through a 2-part system - a mobile app and wearable device that provides a cohesive in-store shopping experience. Methods The cross-platform mobile app, built on Flutter with voice-over capabilities assists the users through the set-up experience. The app utilizes Cloud Firestore as a backend, establishing queries through secured Rest APIs. The wearable device comprises a Jetson Nano, esp32 microcontroller, and a pair of ultrasonic sensors and vibration motors for lateral obstacle detection. Finally, I transfer-learned a MobileNetV2-SSD on custom aggregated/labeled data for grocery item classification and utilized the Tensorflow MIDAS Depth Estimation model to perform monocular obstacle avoidance with depth. Results The final headset weighed in at 2.13 lbs and came to a total of $85 to build. In addition, the grocery classifier achieved a 94.6% average accuracy and the system was able to quickly and accurately detect obstacles with an average of 12 fps and accuracy of 93.33%. Conclusion Moving forward, I plan to distribute the vAssist system to local stores. The project has proven to serve as a proof of concept to support the visually impaired in accomplishing their daily tasks.

Awards (1)

• Second Award of $2,000 $2,000

Competition history

• ISEF 2021 Systems Software · Entry SOFT037

Resources

• ISEF abstract page ↗

Related projects

Closest projects by meaning, across every fair and year in the corpus.