ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation

*Equal contribution.
1Tsinghua University, 2Renmin University of China, 3ShengShu
NeurIPS 2023 (Spotlight)


Score distillation sampling (SDS) has shown great promise in text-to-3D generation by distilling pretrained large-scale text-to-image diffusion models, but suffers from over-saturation, over-smoothing, and low-diversity problems. In this work, we propose to model the 3D parameter as a random variable instead of a constant as in SDS and present variational score distillation (VSD), a principled particle-based variational framework to explain and address the aforementioned issues in text-to-3D generation. We show that SDS is a special case of VSD and leads to poor samples with both small and large CFG weights. In comparison, VSD works well with various CFG weights as ancestral sampling from diffusion models and simultaneously improves the diversity and sample quality with a common CFG weight (i.e., 7.5). We further present various improvements in the design space for text-to-3D such as distillation time schedule and density initialization, which are orthogonal to the distillation algorithm yet not well explored. Our overall approach, dubbed ProlificDreamer, can generate high rendering resolution (i.e., 512x512) and high-fidelity NeRF with rich structure and complex effects (e.g., smoke and drops). Further, initialized from NeRF, meshes fine-tuned by VSD are meticulously detailed and photo-realistic.

Generated Textured Meshes

Michelangelo style statue of dog reading news on a cellphone.

A delicious croissant.

An elephant skull.

A blue tulip.

A small saguaro cactus planted in a clay pot.

A pineapple.

A snail on a leaf.

A 3D model of an adorable cottage with a thatched roof.

A sliced loaf of fresh bread.

A tarantula, highly detailed.

Generated NeRFs

Inside of a smart home, realistic detailed photo, 4k.

Small lavender isometric room, soft lighting, unreal engine render, voxels.

A DSLR photo of a hamburger inside a resturant.

A DSLR photo of an icecream sundae inside a shopping mall.

High-fidelity generated NeRFs.

Complex generated NeRFs.

Diverse Results

A highly detailed sand castle.

A hotdog in a tutu skirt.


      title={ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation},
      author={Zhengyi Wang and Cheng Lu and Yikai Wang and Fan Bao and Chongxuan Li and Hang Su and Jun Zhu},
      journal={arXiv preprint arXiv:2305.16213},