TionAI: Understanding Human Emotion through an Ensemble of Convolutional Neural Networks for better AI-Human Interaction

Overview

Objectives/Goals To create a model that can understand human emotion in such a way that it essentially "feels" emotion just like humans do. Emotion is a key enabler of natural human interaction so AI understanding this idea would not only enable more proactive computers which can interact with humans just like other humans do, but it would also enable AI to better diagnose mental health issues like depression through this same mean of interaction. Methods/Materials Devised an ensemble (an interconnected group) of 3 convolutional neural networks (CNNs) to understand various aspects of emotion in different visual settings (images). One understands emotion in backgrounds/setting (EnvoNet), another understands emotion in objects/foreground (SubjectNet), and the last one synthesizes the decisions from the other 2 CNNs by assigning weightage to each CNN in the final decision made (DeciderNet). Utilized the Wheel of Emotion, an idea that contrasts and characterizes emotions, to derive emotion categories for the ensemble to understand. Wrote a Google Image query script to recursively search through trees of search suggestions to compile datasets for each CNN separated into emotion categories. Trained CNNs through transfer learning, calculating bottleneck values through the penultimate layer of a given pretrained model to gain insights on how to change network weights and biases. Designed a method for testing optimal loss functions and optimizers to yield maximal training results. All datasets split into train, validation, and test (80-10-10) to compute true ensemble accuracy. All training was done using Python 2.7 and TensorFlow, Keras, and other stock math libraries on an NVIDIA GPU. Results Created a network ensemble with validation accuracy of 90.2% overall, exceeding expectations of standard CNNs in such a task. The AI started understanding ideas it wasn't taught through training as well (i.e. associating knives, guns, etc. to various negative emotions). Conclusions/Discussion I trained an ensemble of CNNs to gain visual understanding of human emotion, through transfer learning and various optimization techniques. Performing this approach over an ensemble proved to be effective, and improvements to data collection and bias can further improve ensemble accuracy. The AI is almost indistinguishable from humans in understanding emotion in various images.

Awards (1)

• Honorable Mention

Competition history

• CSEF 2018 Mathematical Sciences · Entry S1519

Resources

• CSEF project summary ↗

Related projects

Closest projects by meaning, across every fair and year in the corpus.

Browse more like this

• All Mathematics projects
• CSEF Mathematics projects
• Every CSEF 2018 project