SoundScape: Real-Time 3D Sound Localization and Classification with Sensory Substitution for the Deaf and Hard of Hearing

Overview

Current devices geared towards the deaf and hard of hearing, such as hearing aids, struggle to localize and transmit sounds to those with severe hearing impairments. Advanced devices, like cochlear implants, are invasive and cost $30,000 to $50,000. Devices that classify sounds, such as home alert systems, are not suited for mobile use and recognize a limited number of noises. Our goal was to convey the directionality, pitch, amplitude, sound classification, and speech recognition of multiple sound sources to those with hearing impairments through a low-cost device. We localized sounds with the SRP-PHAT-HSDA algorithm calculated on incoming audio captured by a 6-microphone array. We separated sounds through Geometric Source Separation (GSS) and beamforming, allowing us to isolate 4 audio sources. We classified each source in real-time using a stacked generalization ensemble trained on an augmented audio dataset. Background noise is filtered through dynamic noise reduction for each source. The direction along with filtered amplitude and frequency are transmitted to the user through vibration motors in a wearable device worn around the shoulders while classification and speech recognition are displayed on a watchOS app. SoundScape can reliably localize four sounds under 15.49 degrees of error, classify sounds with 93.4% accuracy, and operate under 0.2 seconds of latency. The entire device costs $60 to manufacture with inexpensive TPU filament. SoundScape is the first device assistive listening device to separate, localize, and classify multiple sound sources with the potential to protect 466 million people with hearing loss worldwide.

Awards (2)

• Second Award of $2,000 $2,000
• Acoustical Society of America: Second Award of $1000, plus students School will be awarded $100 and Mentor awarded $250. $1,000

Competition history

• ISEF 2021 Robotics and Intelligent Machines · Entry ROBO089T

Resources

• ISEF abstract page ↗

Related projects

Closest projects by meaning, across every fair and year in the corpus.