A Novel Approach to Facial Emotion Recognition Using a CNN and Frame-by-Frame Analysis with Pipe Framework to Aid Students with Autism

Overview

People with autism spectrum disorder (ASD) have difficulty recognizing and comprehending emotions from others’ facial expressions. This inhibits their development of empathy and affects their social interactions. A software package that recognizes emotions in real-time would aid people with ASD. This research tested a new approach using a computer vision (CV) function to split video streams into frame-by-frame images and feeding an ML function the images in Linux pipe framework to detect emotions. First, a CV function was created to take either BSON or FFmpeg-encoded video input, split it into frames, and output both the frame-by-frame images and the original video input. Then a convolutional neural network (CNN) was designed to recognize facial emotion from both the frame-by-frame images and streaming videos. The CNN had 2 convolution layers, a two-route convolution system with max pooling, a third universal convolution layer with a new Leaky ReLU activation function, global average pooling, and a SoftMax affine layer. It was trained with square grayscale images to avoid bias and confounding variables and was tested with live video feed, playback, and frame-by-frame images. The results were compared for the different input types. The model was successful with an average accuracy of 78% for static images, 62% for playback, and 59% for the live video feed. The research also showed 15% lower accuracy for females, and 33% lower accuracy for people of color due to lack of training data in these categories.

Competition history

• ISEF 2020 Robotics and Intelligent Machines · Entry ROBO008

Resources

• ISEF abstract page ↗

Related projects

Closest projects by meaning, across every fair and year in the corpus.