Real-Time 3D Human Tracking and Pose Construction Using Millimeter-Wave Radar Systems

A while ago, I visited my aunt upon her receiving surgery. The first thing I noticed was all the specialized hardware and bulky wires attached to her, used to monitor her vitals (blood pressure, heart rate, temperature). This made me wonder how she would travel around the hospital when necessary. Especially due to the limited hospital personnel regarding the pandemic today, if they go to the bathroom, and they trip, it will be too late for the doctors and the nurses to notice that patient. It turns out that the patient actually has to notify the doctor or nurse before going so the nurse can disconnect all the wires.

There are still numerous problems witnessed with our current monitoring system. Falls are a common and devastating complication witnessed throughout hospital patients. Epidemiological studies have shown that falls occur at a rate of 3-5 per 1000 bed days, which shows just how often these types of falls occur. In a study conducted in 2006, there were over 300,000 patients that witnessed a fall, 26% of them getting injured. The Agency for Healthcare and Research Quality estimates that now (11 years after), over 700,000 to 1 million hospitalized patients fall each year. More than 1/3 of in-hospital falls result in serious injury, these including mainly fractures, head trauma, and possibly death. These falls can not only hurt the patient, but also increase our healthcare.

A potential solution I identified as being able to construct human poses using certain systems to aid in detecting the position of patients. I myself designed SmartApparel: A Low-Cost Hospital Gown for Better Patient Monitoring Systems, which included necessary vitals, along with a 9-Axis DOF (Degrees of Freedom) sensor to measure the patient's position.

Traditional approaches to estimate human pose are based on cameras or wearable sensors.

Although these techniques have achieved impressive recognition accuracy, they are plagued with illumination, privacy, and intrusive user experience issues.

In recent years, millimeter-wave radar technology has emerged as a promising technique that can address the limitations of WIFI, wearable, and camera-based systems. Since they generate and transmit RF signals towards the target, they can maintain a robust operation under poor lighting and weather conditions.

Some efforts have been made to construct human pose using human skeletons by mmPose, mmMesh, RF-Pose3D, and RF-Avatar. However, there is no approach to date that can produce realistic 3D human poses in unconstrained settings while dealing with these challenges.

Therefore, I formulated my research to solve a few of the challenges to accurately construct human poses.

• A low-cost and low-power millimeter-wave radar system that can accurately reconstruct human pose and movements using 3D point cloud data.

• A novel deep Graph Neural Network (GNN) that can take a raw time-series point cloud with irregular length and random order to predict the spatial coordinates of human pose.

• The system should be near real-time to predict the 3D shape, size, and form. The design should be commercially deployable without any specialized hardware.

Related Research

Traditional approaches for estimating human pose are based on cameras or wearable sensors. Although these techniques have achieved impressive recognition accuracy, they are plagued with illumination, privacy, and intrusive user experience issues.

Some efforts have been made to construct human poses using the human skeleton by mmPose, mmMesh, RF-Pose3D, and RF-Avatar. However, there is no approach to date that can produce a realistic, 3D human pose in general, unconstrained settings while dealing with the mentioned challenges.

In recent years, millimeter-wave radar technology has emerged as a promising technique that can address the limitations of Wi-Fi, wearable, and camera-based systems. Since they generate and transmit RF signals towards the target, they can maintain a robust operation under poor lighting and weather conditions. They also address privacy concerns since mmWave radar signals do not involve any video images or facial information.

Introduction

Human tracking and pose estimation play a vital role in human-machine applications, such as health care, intelligent environments, virtual reality, rescue operations, and marketing.

This study proposes a millimeter-wave real-time human recognition and pose construction system. The system will identify the human shape, form, and behavior irrespective of environments with high accuracy and low latency. The system will locate the radar signal variation caused by the moving subject and analyze the 3D point cloud generated by each moving body part. A novel deep Graph Neural Network (GNN) takes the 3D point cloud structures in the spatial dimension, learns the spatial relations between each 3D point cloud, and predicts the human pose structure and shape.

Discussion

The current human pose construction framework is a step towards better human-computer interactions. mm3DPose was developed to solve the complex problem of estimating human pose using mmWave signals. A deep graph neural network was designed to use the point clouds generated by the mmWave device and create a 3D human pose.

Data collected from performing daily activities is used to train and test the model. The results from measuring the average vertex error and average joint localization error suggests that mm3DPose has achieved accurate, smooth, and high-quality human pose reconstruction in real time.

The performance of mm3DPose was tested in different room environments and under different occlusion settings. The performance results in different room environments and many occlusion scenarios suggest that the system can achieve accurate pose construction independent of the room arrangement or settings.

Furthermore, the performance of mm3DPose was compared with current state-of-the-art existing human pose construction methods. The experimental results show that mm3DPose has outperformed current pose reconstruction methods.

Conclusion

In this study, mm3DPose, a mmWave signal-based system, is proposed to reconstruct human pose in real-time. A deep graph neural network is used to construct human pose using the point cloud generated from the mmWave signals. This model uses the Skinned Multi-Person Linear (SMPL) model to address the sparsity of the point cloud. It also uses hypernodes to connect segments from neighboring point cloud to draw the 3D structures of the missing body segments. The results of this study show that mm3DPose system can accurately predict and reconstruct human pose with high quality. This study also suggests that mm3DPose can potentially be included into various applications like healthcare, smart-homes, rescue operations, virtual reality, and security monitoring.

Acknowledgements

•A special appreciation is extended to my primary advisor, and computer science teacher, Mr. Vito Cangelosi, (Old Bridge High School) for his tremendous support

•I want to extend my deepest gratitude to AJAS for allowing me to share my research