MiniPose: Real-Time 3D Joint Estimation and Reconstruction With LiDAR and Low-Resolution Visual Data

Overview

3D pose estimation, the task of reconstructing the skeletal structure of a human subject, has a plethora of applications in medicine, the animation industry, augmented reality, and much more. MiniPose is a novel, resource-efficient algorithm that can accomplish 3D pose estimation on a mobile device all in real-time. While previous alternatives necessitate costly equipment and considerable computational resources, MiniPose opens the door to decentralized trials for biomechanical testing such as gait analysis, affordable alternatives to motion capture, and a promising increase in skeletal research. The MiniPose prototype was initially a supervised model with temporal convolutions deployed on a phone relying on attachable Inertial Measurement Units (IMUs) that can make calculations in 2-3 seconds. The final product removes the need for invasive/non-invasive sensors by utilizing an inexpensive LiDAR scanning device that can track a human subject at all times and an efficient two-step heatmap regression / 2D to 3D lifting algorithm that drastically reduces the cost and time for 3D pose estimation and positioning. MiniPose can predict 2D joints every 6 milliseconds and reconstruct the skeleton in the 3D in less than .6 seconds on average, with over 90% accuracy for both tasks. Coordinating with the LiDAR (which only takes .16 seconds on average to track the subject), MiniPose accomplishes what large motion capture systems and computationally expensive offline 3D pose estimation algorithms can do with a phone (including a physical component costing fewer than 50 dollars) all in real-time, allowing a fast, accurate, and inexpensive solution for all.

Awards (2)

• Third Award of $1,000 $1,000
• University of Texas at Dallas: Back-up scholarship recipients

Competition history

• ISEF 2023 Biomedical Engineering · Entry ENBM055 · Dallas, Texas, United States

Resources

• ISEF abstract page ↗

Related projects

Closest projects by meaning, across every fair and year in the corpus.