Looking through Walls with Artificial Intelligence: An Innovative Solution for Real-Time Retrieval of the Human Figure behind Visual Obstruction

Overview

Overcoming the visual barrier and developing “see-through vision” has been one of mankind’s long-standing dreams. Unlike visible light, Radio Frequency (RF) signals penetrate opaque obstructions and reflect highly off humans. This project created a breakthrough artificial intelligence model that can be trained to reconstruct continuous video of a 15-point human skeleton even through visual occlusion using RF signals. The AI training process adopted a student/teacher learning procedure inspired by the Feynman Technique for Learning. Video frames and RF data were first collected simultaneously using a co-located setup containing an optical camera and an RF antenna array transceiver. Next, the video frames were processed with a computer-vision-based gait analysis “teacher” module to generate ground-truth human skeletons for each frame. Then, the same type of skeleton was predicted from corresponding RF data using a “student” deep-learning model consisting of a Residual Convolutional Neural Network (CNN), Region Proposal Network (RPN), and Recurrent Neural Network with Long-Short Term Memory (RNN-LSTM) that 1) extracted spatial features from RF images, 2) detected all people present in a scene, and 3) aggregated information over multiple time-steps, respectively. After reducing error between teacher-provided ground-truths and student-created predictions over thousands of training iterations, the model was shown to be capable of accurately and completely predicting the pose of any human behind visual obstruction solely using RF signals. Primary academic contributions include the novel many-to-many imaging methodology, unique RPN/RNN-LSTM integration, newly proposed objective function, and original training pipeline. A simulator was also created.

Awards (6)

• Intel ISEF Best of Category Award of $5,000 $5,000
• First Award of $3,000 $3,000
• Association for the Advancement of Artificial Intelligence: First Award of $1,500 $1,500
• Office of Naval Research on behalf of the United States Navy and Marine Corps: The Chief of Naval Research Scholarship Award of $10,000 $10,000
• National Security Agency Research Directorate : First Place Award "Cyber Pioneer" of $1000 $1,000
• Association for Computing Machinery: Fourth Award of $500 $500

Competition history

• ISEF 2019 Robotics and Intelligent Machines · Entry ROBO059

Resources

• ISEF abstract page ↗

Related projects

Closest projects by meaning, across every fair and year in the corpus.