🔥The 9th Round of Easy Loan, Earn $40 Reward is in progress❗️ ⏰ Promotion Period: January 15th - Feburary 15th, 2025 👉 Register now and check more details at gate.io/campaigns/358

BIG-Bench Extra Hard Google DeepMind introduces BIG-Bench Extra Hard (BBEH), a new benchmark designed to push the boundaries of LLM reasoning evaluation. "BBEH replaces each task in BBH with a novel task that probes a similar reasoning capability but exhibits significantly

Happy to see a flavor 🧂 of Compositional GSM in this work Comp GSM: arxiv.org/abs/2410.01748 BBEH: arxiv.org/abs/2502.19187

“It’s been only <N> days since <Company> released <Model> and people can’t stop building with it” is sooooo cliche! Almost half of my timeline every day! Are AI influencers all using the same LLM for writing their tweets?

Glad to see our BIG-Bench Extra Hard paper featured.

Happy to see that Linguini, our benchmark for language-agnostic linguistic reasoning, has been included in DeepMind’s BIG-Bench Extra Hard (BBEH). Linguini remains challenging for reasoning models, being one of only two (hard) tasks where o3-mini doesn't show massive gains.

New Paper Alert!📄 "It’s better to be sparse than to be dense" ✨ We explore how to steer LLMs (like Gemma-2 2B & 9B) by modifying their activations in sparse spaces, enabling more precise, interpretable control & improved monosemanticity with scaling. Let’s break it down! 🧵

The evaluation code is now available at: github.com/google-deepmin… Also consider submitting to our leaderboard: github.com/google-deepmin…

Checkout our latest Gemma models, you'll love them :) Also checkout the results on our BIG-Bench Extra Hard benchmark:

We’ve had several papers rejected primarily because of this rule and because many researchers don’t know about it. A NeurIPS AC even accused us of lying, and an ACL AC literally said “it’s your problem”. Hope Sam Altman’s tweet increases the awareness about this rule.

Excited to be at #ICLR2025 next week! I'm currently on the job market for Research Scientist positions, especially in generative modeling, synthetic data, diffusion models, or responsible AI. Feel free to reach out if you have any openings!

✈️ I will be at ICLR 2026 🇸🇬 to present the following work on LLM reasoning, vision-language understanding, and LLM evaluation w/ uclanlp, UCLA Machine Intelligence (MINT), and Google DeepMind! Come to the poster sessions and say hi 👋 I will be happy to meet folks from

I'm presenting our recent work on "Pitfalls of Memorization" today at ICLR Number #304 at 3pm.. Come say hi! iclr.cc/virtual/2025/p…

New Paper! 📣 RL^V: a unified RL & generative verifier —boosts MATH accuracy by 20% and improves both sequential and parallel test-time scaling ☑️ improves out-of-domain and easy-to-hard generalization ☑️ allows dynamic allocation of compute for harder problems How? 👇🏻

Upon some requests, we now have a BBEH Mini with 460 examples (20 per task) for faster and cheaper experimentation. The set can be downloaded from: github.com/google-deepmin… The results are reported in Table 3 of arxiv.org/pdf/2502.19187

crazy to see a wild research idea I dreamt about for a long time land into this Gemini 2.5 Flash! it's been amazing making this happen with Yi Tay and Quoc Le, and the rest of the team!!!

Great to see that the latest #GeminiDiffusion release benchmarks on our challenging general-purpose reasoning Big Bench Extra Hard dataset! It is now available on HF 🤗: huggingface.co/datasets/BBEH/… Eval code: github.com/google-deepmin…

I'm excited to be speaking at the 1st Workshop on Large Language Models for Cross-Temporal Research at COLM 2025 in Montreal! This workshop is tackling crucial issues around LLMs' understanding of time. Don't miss out! More details & submission deadlines: lnkd.in/eaaAR_z2

I'm the first author on all three of [23, 24, 25]. Not sure if I should be flattered or offended 😄😁😅

Are LLMs good data scientists? Our new work, lead by Ken Gu, provides a detailed analysis.

Reviewed and meta-reviewed by an LLM :)