Dilated Silhouette Convolutional Neural Network: A Novel Deep Learning Framework for Real-time Human Action Recognition

Overview

With a wide range of applications in artificial intelligence (AI), human action recognition is one of the most attractive yet challenging research fields. It has the potential to foster natural human-computer interaction, increase public safety, aid physical therapy, conduct sport analysis, etc. Human action is a spatio-temporal sequence with strong interdependencies between the spatial geometry and temporal dynamics of motion. Many AI methods have been proposed for human action recognition; however, in existing literature, there is a lack of synergy in investigating spatial geometry and temporal dynamics in a joint representation and embedding space. Therefore, I propose a novel dilated Silhouette Convolutional Neural Network (SCNN) for action recognition from a video. In SCNN, spatial geometric information of the moving human is modeled with silhouettes extracted from each frame of the video. The silhouettes are stacked along time and resampled to form a 3D point cloud, a unified spatio-temporal representation of the video action. With dilated silhouette convolutions, SCNN learns co-occurrence features from low-level geometric silhouettes and their temporal dynamics and constructs a unified convolutional embedding space, where spatial and temporal properties are integrated effectively. Geometry-based SCNN, complemented by an image-based approach, significantly improves the distinctiveness of learned features from motion. Experiment results on the public JHMDB, HMDB, and UCF101 datasets demonstrate the effectiveness of my method, outperforming all other state-of-the-art algorithms. I further integrate my SCNN model into a real-world product, an iOS app, to recognize actions and provide real-time coaching in fitness and physical therapy.

Awards (3)

• First Award of $5,000 $5,000
• George D. Yancopoulos Innovator Award
• Association for the Advancement of Artificial Intelligence: First Award of $1,500 $1,500

Competition history

• ISEF 2021 Robotics and Intelligent Machines · Entry ROBO033

Resources

• ISEF abstract page ↗

Related projects

Closest projects by meaning, across every fair and year in the corpus.