🔥The 9th Round of Easy Loan, Earn $40 Reward is in progress❗️ ⏰ Promotion Period: January 15th - Feburary 15th, 2025 👉 Register now and check more details at gate.io/campaigns/358

Excited to be at #NAACL2025 in Albuquerque this week! I'll be presenting "OLMES: A Standard for Language Model Evaluations" (arxiv.org/abs/2406.08446)! Work done with my wonderful collaborators at Ai2 ❤️

Accepted to #ICML2025 ! 🥳

Also accepted to #ICML2025 ! 🤩 See you in Vancouver this summer!

Responsible Reviewing #NeurIPS2025 — TL;DR 1- If you/ your co-author skip your assigned reviews → you wont see your own paper’s reviews. 2- Submit a poor quality review → your paper may be desk‑rejected. 👏 Nice one, NeurIPS! 🔗 blog.neurips.cc/2025/05/02/res…

Current KL estimation practices in RLHF can generate high variance and even negative values! We propose a provably better estimator that only takes a few lines of code to implement.🧵👇 w/ Tim Vieira and Ryan Cotterell code: arxiv.org/pdf/2504.10637 paper: github.com/rycolab/kl-rb

Lucas Beyer (bl16) (((ل()(ل() 'yoav))))👾 rohan anil arxiv.org/abs/2410.15002 This is in the text-to-image domain, and we have some ideas on how to extend this to the text domain We also recently published this: arxiv.org/abs/2504.12459, which connects the number of entities co-occurrences, and "linearity" in model representations:

🆕paper: LLMs Get Lost in Multi-Turn Conversation In real life, people don’t speak in perfect prompts. So we simulate multi-turn conversations — less lab-like, more like real use. We find that LLMs get lost in conversation. 👀What does that mean? 🧵1/N 📄arxiv.org/abs/2505.06120

Excited to share that our SafetyAnalyst paper has been accepted to #icml2025! SafetyAnalyst provides a novel way to determine if some AI behavior would be safe. It’s accurate, interpretable, transparent, and steerable. 1/7

Try cheap model adapters for formatting outputs Don't assume model providers' structured output apis offer best performance

RewardBench 2 is here! We took a long time to learn from our first reward model evaluation tool to make one that is substantially harder and more correlated with both downstream RLHF and inference-time scaling.

🚨 New Paper! 🚨 Guard models slow, language-specific, and modality-limited? Meet OmniGuard that detects harmful prompts across multiple languages & modalities all using one approach with SOTA performance in all 3 modalities!! while being 120X faster 🚀 arxiv.org/abs/2505.23856

I’m thrilled to share RewardBench 2 📊— We created a new multi-domain reward model evaluation that is substantially harder than RewardBench, we trained and released 70 reward models, and we gained insights about reward modeling benchmarks and downstream performance!

The paper is now up on ArXiv! arxiv.org/abs/2506.01937

🚨 Introducing LAQuer, accepted to #ACL2025 (main conf)! LAQuer provides more granular attribution for LLM generations: users can just highlight any output fact (top), and get attribution for that input snippet (bottom). This reduces the amount of text the user has to read by 2

We enabled OLMoTrace for Tülu 3 models! 🤠 Matched spans are shorter than for OLMo models, bc we can only search in Tülu's post-training data (base model is Llama). Yet we thought it'd still bring some value. Try yourself on the Ai2 playground -- playground.allenai.org

i created a gist with some non-default LLM courses: gist.github.com/yoavg/95bbc576…