Generative AI + biomolecular structure is gaining traction again! It's been a year since we introduced NeuralPLexer arxiv.org/abs/2209.15171 to jointly predict and dynamically sample diverse compound(s)-protein complex structures. Glad to see works adopting relevant strategies🧵:

Text understanding with #LLMs is useful but not enough for scientific understanding and discovery. In chemistry, in addition to text, chemical structure is essential to determine the properties of molecules. We have created the first multimodal text-chemical structure model:

🖇 T-Stitch Accelerating Sampling in Pre-trained Diffusion Models with Trajectory Stitching Sampling from diffusion probabilistic models (DPMs) is often expensive for high-quality image generation and typically requires many steps with a large model. In this paper, we introduce

Both Sora and Stable Diffusion 3 adopt diffusion transformers, but do we really need a super large DiT for all sampling steps for generation?🧐 No🙅‍♂️. We found ~40% early timesteps of DiT-XL can be replaced with a 10x faster DiT-S without image quality drop! Introduce

Thrilled that our work on generative AI for dynamic protein-ligand binding is now on the cover of nature Zhuoran Qiao / 乔卓然 Weili Nie Thomas Miller nature.com/articles/s4225…

Efficient Video Diffusion Models via Content-Frame Motion-Latent Decomposition Video diffusion models have recently made great progress in generation quality, but are still limited by the high memory and computational requirements. This is because current video

Thank you AK for showcasing our paper on efficient video diffusion. We propose content-motion latent diffusion model - an efficient video model that utilizes pretrained image diffusion models and a low-dimensional motion latent representation for video generation

Compositional Text-to-Image Generation with Dense Blob Representations Existing text-to-image models struggle to follow complex text prompts, raising the need for extra grounding inputs for better controllability. In this work, we propose to decompose a scene into visual

📢🔥 Compositional Text-to-Image Generation w. BlobGEN Compositionality & modularity are some of the fundamental problems in text-to-image generation. We introduce BlobGEN which breaks image generation into 2 stages: an LLM generates scene layout & a diffusion model renders it.

🔥 Check our new work on controlling camera in video diffusion models. Our approach takes a pretrained model and injects the camera params into it via Plucker embeddings & epipolar attention. arxiv.org/abs/2406.02509 Dejia Xu, Weili Nie, Chao Liu, Sifei Liu, Jan Kautz Atlas Wang

[1/7] 📜 I can finally share that our recent @NVIDIA project DiffUHaul --- A Training-Free Method for Object Dragging in Images has been accepted to #SIGGRAPHAsia2024 🎉. Project Page: omriavrahami.com/diffuhaul/

🔥🔥Our new #SIGGRAPHAsia2024 paper shows how compositional text-to-image diffusion models (BlobGEN) can be used for dragging objects around in an image.

Time to make your diffusion models one step! Excited to share our recent work on Truncated Consistency Models, a new state-of-the-art consistency model. TCM outperforms a previous SOTA, iCT-deep, using more than 2x smaller networks in both one-step and two-step FIDs. Joint work

📢Annoucing EDLM, our brand-new Energy-based Language Model embedded with Diffusion framework! Key results: 1. We (for the first time?) almost match AR perplexity. 2. Significantly improved generation quality. 3. Considerable sampling speedup without quality drop. 🧵1/n

📢 Warped Diffusion Our new #neurips2024 presents a simple approach to turn image2image models into video2video models. arxiv.org/abs/2410.16152 giannisdaras.github.io/warped_diffusi… with: Giannis Daras, Weili Nie, Karsten Kreis , Alex Dimakis , Morteza Mardani, Nik Kovachki

New #NVIDIA paper to make diffusion models better and faster 🚀 Multi-Student Distillation! We distill diffusion models into multiple 1-step students, allowing (a) improved quality by specializing in subsets and (b) improved latency by distilling into smaller architectures. 1/n

🚀Excited to share our f-RAG at #NeurIPS2024, a molecular optimization framework that leverages fragment-level RAG. arxiv.org/abs/2411.12078 with incredible collaborators: Karsten Kreis, Srimukh Prasad Veccham, Meng Liu, Danny Reidenbach, Saee Paliwal, Arash Vahdat, Weili Nie

Recently had several chats on AI4Science alignment/RLHF stuff, and realized that I missed posting our NeurIPS24 work: A brief thread: Aligning Target-Aware Molecule Diffusion Models with Exact Energy Optimization Paper: openreview.net/forum?id=EWcvx… Code: github.com/MinkaiXu/AliDi… 1/n

Tired of slow diffusion models? Our new paper introduces f-distill, enabling arbitrary f-divergence for one-step diffusion distillation. JS divergence gives SOTA results on text-to-image! Choose the divergence that suits your needs. Joint work with Weili Nie Arash Vahdat 1/N

🎉Thrilled to share my internship work with the @NVIDIA GenAIR team (accepted to #CVPR2025): BlobGEN-Vid: Compositional Text-to-Video Generation with Blob Video Representations! 🚀BlobGEN-Vid is a model-agnostic framework that delivers: - SOTA layout controllability - Enhanced