My awesome student James Y. Huang just received an outstanding paper award at #EMNLP2023! He is looking for summer research intern. Please interview him.

DeAL Decoding-time Alignment for Large Language Models paper page: huggingface.co/papers/2402.06… Large Language Models (LLMs) are nowadays expected to generate content aligned with human preferences. Current work focuses on alignment at model training time, through techniques such

🤨 Frustrated when LLMs can give different or even wrong answers because you phrased your instruction differently? 🚀 Introducing Contrastive Instruction Tuning (CoIN) - our solution to make models less sensitive to the exact phrasing of instructions! (arxiv.org/abs/2402.11138)

Great news to wrap up this school year with my very first first-author paper: CoIN got accepted at #ACL2024 Findings! (github.com/luka-group/CoIN) 🌟Many thanks to my collaborators Fei Wang, James Y. Huang, Wenxuan Zhou, Fan Yin, Aram Galstyan and advisor 🌴Muhao Chen🌴

Can Text-to-Image models understand common sense? 🤔 Can they generate images that fit everyday common sense? 🤔 tldr; NO, they are far less intelligent than us 💁🏻‍♀️ Introducing Commonsense-T2I 💡 zeyofu.github.io/CommonsenseT2I/, a novel evaluation and benchmark designed to measure

With the fast-growing interest in multi-image understanding, where can we find a high-quality, comprehensive & robust benchmark? A group of talented students have created ᴍᴜɪʀʙᴇɴᴄʜ, a long-awaited multi-image benchmark, with 2.6k multi-choice Qs and over 10k images. (1/6)

🌟 𝐌𝐮𝐥𝐭𝐢𝐦𝐨𝐝𝐚𝐥 𝐃𝐏𝐎🌟 🔍 DPO over-prioritizes language-only preference 🚀 Introducing mDPO: optimizes image-conditioned preference 🏆 Best 3B MLLM with reduced hallucination, beats LLaVA 7/13B with DPO Collaboration with Microsoft Research huggingface.co/papers/2406.11…

Can GPT-4o and Gemini-Pro handle 𝐦𝐮𝐥𝐭𝐢𝐩𝐥𝐞 𝐢𝐦𝐚𝐠𝐞𝐬? Introducing MuirBench: A Comprehensive Benchmark for Robust Multi-image Understanding. 🌐 Explore here: muirbench.github.io 📄 Paper: arxiv.org/abs/2406.09411 📊 Data: huggingface.co/datasets/MUIRB…

🚨 DPO works well on LLMs, but what happens when we apply it to VLMs/MLLMs? 🚀 Excited to introduce our #EMNLP2024 work on DPO for Multimodal LLMs! We identify the unconditional preference issues (where the model 𝐨𝐯𝐞𝐫𝐥𝐨𝐨𝐤𝐬 𝐯𝐢𝐬𝐮𝐚𝐥 𝐜𝐨𝐧𝐭𝐞𝐱𝐭), leading to severe

🚀 Excited to share MetaScale, our latest work advancing LLM reasoning capabilities! MetaScale empowers GPT-4o to match or even surpass frontier reasoning models like o1, Claude-3.5-Sonnet, and o1-mini on the challenging Arena-Hard benchmark (lmarena.ai). Additionally, MetaScale

🎉 Excited to share that our paper, "MuirBench: A Comprehensive Benchmark for Robust Multi-image Understanding", will be presented at #ICLR2025!​ 📅 Date: April 24 🕒 Time: 3:00 PM 📍 Location: Hall 3 + Hall 2B #11 MuirBench challenges multimodal LLMs with diverse multi-image