🔥The 9th Round of Easy Loan, Earn $40 Reward is in progress❗️ ⏰ Promotion Period: January 15th - Feburary 15th, 2025 👉 Register now and check more details at gate.io/campaigns/358

CDS Prof. Kyunghyun Cho (Kyunghyun Cho) has published two new papers, urging a reevaluation of how progress in AI is measured. Are we advancing or just repeating history? Learn more: nyudatascience.medium.com/separating-hyp…

If you're at #NAACL2024 come say hi at the poster session!! 😁

Do LLMs exploit imperfect proxies of human preference in context? Yes! In fact, they do it so severely that iterative refinement can make outputs worse when judged by actual humans. In other words, reward hacking can occur even without gradient updates! w/ He He,

SOTA models often use bidirectional transformers for non-NLP tasks but did you know causal transformers can outperform them even on tasks without a causal structure? Our recent work shows causal transformers learn circuits bidirectional ones can't, leading to better performance!

There are many algorithms for constructing pre-training data mixtures—which one should we use? Turns out: many of them fall under one framework, have similar issues, and can be improved with a straightforward modification. Introducing Aioli! 🧄 1/9

So you want a good pretraining data mix🧑‍🍳, but which data mixing algorithm do you pick? DoGE, DoReMi, Skill-it, grid searching proportions… 😵‍💫 It turns out that these algorithms are all special cases of Linear Mixing Optimization (LMO), our new data mixing framework! 🧵

If scaling no longer makes economic sense, what does that mean for research?? Will we see more work on architecture and fundamentals again? Or, will the current spread of topics remain unchanged? 🤔

Finding good data mixtures for LLM training can be tricky - Aioli provides a unified framework to construct pre-training data mixtures. Talk to the authors Mayee Chen Michael Hu Nicholas Lourie Kyunghyun Cho Chris Re hazyresearch directly here!

New Anthropic research: Adding Error Bars to Evals. AI model evaluations don’t usually include statistics or uncertainty. We think they should. Read the blog post here: anthropic.com/research/stati…

I missed this great paper when it came out last year! TL;DR: Prediction errors from models trained with different random seeds become independent as training converges---at least for the image classification tasks they consider. This finding has important implications if you're

Anthropic put out a great primer on statistical methods for LLM evals by Evan Miller. Check out his blog too! He's written gems on A/B testing and other topics---just make sure you don't mind losing an afternoon like I did when I first came across it! 😆 evanmiller.org

Can we predict emergent capabilities in GPT-N+1🌌 using only GPT-N model checkpoints, which have random performance on the task? We propose a method for doing exactly this in our paper “Predicting Emergent Capabilities by Finetuning”🧵

A great idea by Charlie Snell: Use finetuning to predict where zero-shot capabilities emerge. This lets you experiment at a smaller scale. The more finetuning data you have, the smaller of a model you can use. Here's how I think about it: a one-time cost collecting data saves you

📢 today's scaling laws often don't work for predicting downstream task performance. For some pretraining setups, smooth and predictable scaling is the exception, not the rule. a quick read about scaling law fails: 📜arxiv.org/abs/2507.00885 🧵1/5👇