Revival and Restoration of Chinese Traditional Puppetry Videos Using Motion Video and Action Recognition Technology
ISEF · 2024 Technology Enhances the Arts
Overview
To tackle the decline of the art of shadow puppetry, this project provides a systematic way of preserving and promoting the art form. The procedure includes repairing fragmented recordings, capturing key points from the processed videos, and perform the content of the videos in real life using a mechanical structure. This project first endeavors to introduce an innovative AI-driven video generation technique specializes at transforming static images into continuous and dynamic video sequences. The model architecture comprises an image pair generator leveraging StyleGAN2 to produce sequential images; motion codes derived from the Jacobian matrix and low-rank factorization of the generator; and a video generator that iteratively manipulates image pairs using the motion codes. Results show that this model can be effectively used to repair shadow puppetry videos, and can serve as a novel way of art creation as well. Using AI action recognition technology on the repaired videos, key-points controlling the shadow puppetry characters are extracted from the repaired or generated videos. Then, a mechanical structure made up of three cross-slide tables controlling puppetry characters according to the obtained key-points is designed. With that, we are able to re-enact the way of performance of shadow puppetry in real life. This provides a real and beautiful shadow puppetry experience, while the content of the performance is the same as generated videos. In this way, a new, easy to use and advanced way of preserving, creating and performing the art of shadow puppetry is obtain. It not only can be used to help promote the beauty of the art and gain public attention and love, but also serve as an important step of modernization of this time-honored art.
Competition history
- ISEF 2024
Resources
Related projects
ISEF · 2025
RestART: An AI-Driven Restoration System for High-Fidelity in Artwork Revival Across Multiple Surfaces, Damage Types, and Eras Through Metadata-Enriched, Style-Conscious, Explainable, and Interactive Artist Approaches
ISEF · 2021
Magic-Beijing-Mirror: Facilitating Beijing Opera via a Real-Time Interactive Opera Making up System
ISEF · 2023
The Story of the Stone: A Novel Rubbing Restoration Method Using Generative Adversarial Network
ISEF · 2026
Kintsugi 4D: Image-Based Kintsugi-Aware Framework for Fragment-Level 3D Point-Cloud Reconstruction and Chronicle Integration
Closest projects by meaning, across every fair and year in the corpus.
Source: Regeneron International Science and Engineering Fair