Diffusion-Based Conditional Mesh Synthesis for Rapid 3D Visual Design
ISEF · 2024 Technology Enhances the Arts First Award
Overview
As digital design has become ubiquitous over the last few decades, 3D creations have become much more advanced through Computer-Aided Design (CAD). However, this approach can be tedious, time-consuming, and inaccessible, especially when creating the kinds of smooth and curved designs necessary in art. To accelerate 3D artwork, this research presents CloudGen, a revolutionary text-to-3D printing pipeline to create 3D models from natural language prompts, allowing artists to shape objects using text. To this end, a Latent Diffusion Model (LDM) was implemented to generate print-ready 3D structures based on text inputs. First, an encoder-decoder mechanism transforms point clouds into functions which parametrize 3D shapes. Then, using pre-trained CLIP embeddings to convert text prompts into numerical representations, a vision transformer creates such 3D shapes, repeatedly iterating upon the object embedding towards the shape designated by the prompt. Once the initial synthesis is complete, the shape generated is smoothed using the bilateral filter algorithm, thereby removing any inconsistencies and instabilities. Further, to enable users to modify their generations, a novel editing framework was developed to retrace diffusion steps and incorporate feedback prompts. The end-to-end generation architecture has been implemented in a user-friendly web interface paired with a virtual reality-powered design viewer. Both quantitative and qualitative experiments validate the efficacy of the proposed text-to-3D printing pipeline. With this research, CloudGen establishes the foundation for human-in-the-loop creation of 3D art pieces.
Awards (3)
- First Award of $5,000 $5,000
- Air Force Research Laboratory on behalf of the United States Air Force: Glass trophy and USAF medal for each recipient
- Air Force Research Laboratory on behalf of the United States Air Force: First Award of $750 in each Regeneron ISEF Category, $750
Competition history
- ISEF 2024
Resources
Related projects
ISEF · 2022
3D AttnGAN for Text to Voxel Generation
ISEF · 2025
Symmetry-Preserving Variational Autoencoder and Latent Space Rectified Flow Diffusion for Accelerated Materials Discovery With Geometric Graph Neural Networks and Integrated Property Network Multi-Layer Perceptron
ISEF · 2025
Procedural 3D Object Generation for Novel Contrastive Learning in Computer Vision
ISEF · 2026
Spatial Transcriptomics-Conditioned Latent Diffusion Models for Synthetic Histopathology Tissue Patch Generation
Closest projects by meaning, across every fair and year in the corpus.
Source: Regeneron International Science and Engineering Fair