IllumiNeRF lets you relight objects in 3D. Instead of the classical inverse rendering approach — disentangling the object geometry, materials, and lighting — we use a relighting diffusion model to relight each input image and distill the relit samples into 3D by optimizing a

This work led by Haian Jin is really nice. It takes text-to-image models and teases out their capability to light objects in a controllable way, much like Zero123 does for camera viewpoint. I'm really surprised that conditioning on environment maps can work this well!

There is a lot happening on the lighting front these days! Another very nice recent paper on relighting is called IllumiNeRF. x.com/xmzhao_/status…

Our Machine Learning Research (MLR) team at #Apple is seeking a passionate AI resident to conduct research on multi-modal generative models (vision, 3D, language, audio) and to explore effective control mechanisms for these models. Application details: jobs.apple.com/en-us/details/…

Excited to share that I'll be joining University of California at Irvine as a CS faculty in '25!🌟 Faculty apps: Krishna Murthy, Zhuang Liu & I share our tips: unnat.github.io/notes/Hidden_C… PhD apps: I'm looking for students in vision, robot learning, & AI4Science. Details👇

look what we've been cooking at world labs. i'm really proud of the team -- this is the product of everyone's hard work and enthusiasm. we have a long way to go from here, but i'm excited about about the future. if you'd like to build that future with us, we're hiring!

I will be at #NeurIPS2024 this week in Vancouver. Very excited to discuss the future of generative video models and 3D. We will also be presenting illuminerf.github.io and cat3d.github.io. Looking forward to meeting everyone. Please reach out if you’d like to meet

IllumiNeRF is #NeurIPS2024 bound! 🌟 Updated ArXiv with more results & discussions. Catch Philipp Henzler at #NeurIPS to learn more! Paper: arxiv.org/abs/2406.06527 Webpage: illuminerf.github.io Location: East Exhibit Hall A-C #1404 Time: Fri 13 Dec 11 a.m. PST — 2 p.m. PST

Congrats to Rex Cheng for impressive work on generating audio from video and/or text.

What matters for runtime optimization in Vision Language Models (VLMs)? Vision encoder latency 🤔? Image resolution 🤔? Number of visual tokens 🤔? LLM size 🤔? In this thread, we break it all down and introduce FastVLM — a family of fast and accurate VLMs. (1/n 🧵)

We are looking for a summer research intern to work on improving TarFlow at Apple. You will be working with myself and a great group of researchers, Jiatao Gu Preetum Nakkiran David Berthelot etc. If interested, send your CV to szhai at apple.com by this week.

We are looking for an intern to join MLR at Apple ASAP. You will be working with me, Yuyang Wang and other scientists in the team on domain-agnostic generative modeling at scale. If you are interested please reach out to mbautistamartin at apple by the end of this week.

Excited to share code & models for FastVLM — our blazing-fast Vision-Language Model appearing at #CVPR2025 Run it on-device with inference code optimized for Apple Silicon using #mlx. Code: github.com/apple/ml-fastv… Updated paper & results coming soon. Stay tuned! 👀

We have an open position at Apple MLR to work scalable and efficient generative models that perform across diverse data domains—including images, 3D, video, graphs, etc. We care deeply about simplifying modeling pipelines, developing powerful and scalable training recipes.

Come work with us! The Machine Learning Research (MLR) team at Apple is seeking a passionate AI researcher to work on Efficient ML algorithms: jobs.apple.com/en-us/details/…

Is your AI keeping Up with the world? Announcing #NeurIPS2025 CCFM Workshop: Continual and Compatible Foundation Model Updates When/Where: Dec. 6-7 San Diego Submission deadline: Aug. 22, 2025. (opening soon!) sites.google.com/view/ccfm-neur… #FoundationModels #ContinualLearning

🌟Explore key insights from the FastVLM project (real-time vision-language model) in this blog post: machinelearning.apple.com/research/fast-…

The latest MLX has a CUDA back-end! To get started: pip install "mlx[cuda]" With the same codebase you can develop locally, run your model on Apple silicon, or in the cloud on Nvidia GPUs. MLX is designed around Apple silicon - which has a unified memory architecture. It uses

🚀🤗FastVLM models are now on HuggingFace! Enabling real-time VLM applications at up to 85x faster than prior work and 3.4x smaller. huggingface.co/collections/ap… Checkout a cool demo on HuggingFace: huggingface.co/spaces/apple/f… Huge thanks to the amazing folks at HuggingFace! #Apple MLR

NEW: Apple releases FastVLM and MobileCLIP2 on Hugging Face! 🤗 The models are up to 85x faster and 3.4x smaller than previous work, enabling real-time VLM applications! 🤯 It can even do live video captioning 100% locally in your browser (zero install). Huge for accessibility!