🔥The 9th Round of Easy Loan, Earn $40 Reward is in progress❗️ ⏰ Promotion Period: January 15th - Feburary 15th, 2025 👉 Register now and check more details at gate.io/campaigns/358

Excited to share our recent work at #NeurIPS2023 on the nature of Simplicity Bias (SB) in 1-Hidden Layer Neural Networks (NNs) with Jatin Batra Prateek Jain Praneeth Netrapalli. SB is known to be one of the reasons behind brittleness of neural networks towards distribution shift (1/5)

It’s an honor to be part of the 2024 cohort of Kempner Institute graduate fellows! Excited for what lies ahead :)

Introducing ReSearch: An iterative self-reflection algorithm that enhances LLM's self-restraint abilities: • Encouraging abstention when uncertain • Producing accurate, informative content when confident Result: Significant accuracy boost for Llama2 7B Chat and Mistral 7B! 🚀

Are you attending #ICML2024? Be sure to check out our posters, workshops & presentations from the #KempnerInstitute community! Ada Fang Blake Bordelon ☕️🧪👨‍💻 Cengiz Pehlevan Riley Simmons-Edler Naomi Saphra Kanaka Rajan David Brandfonbrener Rosie Zhao Eran Malach Ryan P. Badman Marinka Zitnik

1/n Introducing SOAP (ShampoO with Adam in the Preconditioner's eigenbasis): A deep learning optimization algorithm that applies Adam in Shampoo's eigenbasis. SOAP outperforms both AdamW and Shampoo in language model pretraining.

Combining SOAP and Muon: nikhilvyas.github.io/SOAP_Muon.pdf and some rough thoughts on interesting future directions.

Ever looked at LLM skill emergence and thought 70B parameters was a magic number? Our new paper shows sudden breakthroughs are samples from bimodal performance distributions across seeds. Observed accuracy jumps abruptly while the underlying accuracy DISTRIBUTION changes slowly!

Meet the recipients of the 2024 ACM A.M. Turing Award, Andrew G. Barto and Richard S. Sutton! They are recognized for developing the conceptual and algorithmic foundations of reinforcement learning. Please join us in congratulating the two recipients! bit.ly/4hpdsbD

🚨 New preprint! TL;DR: Backtracking is not the "holy grail" for smarter LLMs. It’s praised for helping models “fix mistakes” and improve reasoning—but is it really the best use of test-time compute? 🤔

How does RL improve performance on math reasoning? Studying RL from pretrained models is hard, as behavior depends on choice of base model. 🚨 In our new work, we train models *from scratch* to study the effect of the data mix on the behavior of RL. arxiv.org/abs/2504.07912

Excited to be attending 🇸🇬#ICLR2025! My DMs are open, please reach out to chat about LLM reasoning/optimization/training dynamics! Will be presenting a study on diagonal preconditioning optimizers for LLM pretraining (arxiv.org/abs/2407.07972) and SOAP (arxiv.org/abs/2409.11321)

Excited to announce MOSS, our ICML workshop focused on discoveries at small scale! We believe there's tremendous potential & creativity in research done with limited resources and would love to hear your ideas. The submission (due May 22nd) can literally be a Jupyter notebook! :)

Announcing the first workshop on Foundations of Post-Training (FoPT) at COLT 2025! 📝 Soliciting abstracts/posters exploring theoretical & practical aspects of post-training and RL with language models! │ 🗓️ Deadline: May 19, 2025

Adam is similar to many algorithms, but cannot be effectively replaced by any simpler variant in LMs. The community is starting to get the recipe right, but what is the secret sauce? Robert M. Gower 🇺🇦 and I found that it has to do with the beta parameters and variational inference.