Voicemath: A Calculator to Convert Commonly Spoken Mathematical Speech Into Equations Using Natural Language Processing

Overview

Calculators are deeply integrated into math education and in real life, however there has yet to be an accessible calculator for individuals with limited hand usage. While attempts have been made to create a speech based calculator program, it has been difficult due to the difference in spoken mathematics, which we use to “talk out” mathematics such as equations, and written mathematical texts. Spoken math is much less rigid than in text: sentence structures vary, numbers or variables are left out, and some math syntax, such as brackets, aren’t spoken at all. Voicemath is a program that lets users transcribe everyday mathematical speech into written mathematics. Instead of developing a separate, abstruse syntax, it uses Automatic Speech Recognition (ASR) and Intent Classification to dynamically infer the user’s intent and mathematical equation, then displays it on a screen. ASR was achieved by fine-tuning a speech-to-text model using a self-created dataset of approximately 1000 audio files consisting of mathematical speech and corresponding math sentences. Intent classification is used afterwards to tokenize and tag relevant mathematical text. Voicemath has demonstrated a high accuracy in transcribing mathematical speech, with a Word Error Rate (WER) of 10%, which is twice as lower than that reported in previous studies. Unlike other existing programs, Voicemath allows users to omit details and input mathematical speech that deviates from standard patterns. Voicemath is also integrated into a calculator to transcribe speech into text. It enhances the accessibility and usability of math programs, which have traditionally been geared towards individuals who are physically able and possess prior mathematical knowledge.

Competition history

• ISEF 2024 Technology Enhances the Arts · Entry TECA012T · Los Angeles, California, United States

Resources

• ISEF abstract page ↗

Related projects

Closest projects by meaning, across every fair and year in the corpus.