🔥The 9th Round of Easy Loan, Earn $40 Reward is in progress❗️ ⏰ Promotion Period: January 15th - Feburary 15th, 2025 👉 Register now and check more details at gate.io/campaigns/358

This is happening now! I will be poster #913 presenting this work. Happy to chat if you are interested!

📡Join us at the 2nd workshop on Mathematics of Modern Machine Learning (M3L) at #NeurIPS2024! sites.google.com/view/m3l-2024/ Submission deadline: September 29, 2024

We've extended the #M3L submission deadline to October 1st AoE to align with ICLR timelines. We look forward to your work!

Exciting new work led by amazing Kaiyue Wen on theoretical justification for the recent popular WSD schedule! This is based an interesting and novel assumption of training loss called "River Valley", which is useful to explain hidden progress in large learning rate training.

Hope everyone had fun at the 2nd workshop of M3L! Many thanks to the speakers, authors, reviewers, and participants for making this workshop a success. We had a full house again, and we hope to see you next year! 💡

Can we quantify the effect of learning rate schedules? Empirically, what's the best schedule for LLM pretraining? 🚀Excited to share our ICLR paper! arxiv.org/abs/2503.12811 With ≤3 runs, you can fit our empirical law and optimize your schedule—a WSD-like schedule is the best!

SVRG is popular in theoretical optimization, but it has not been widely adopted to train large neural networks. In our ICLR work “A Coefficient Makes SVRG Effective”, we show that adding a coefficient helps SVRG optimize deep neural networks. arxiv.org/abs/2311.05589

Don't miss the poster presentation for this by Nishanth Dikkala at #ICLR2025 tomorrow to learn more about our work on looped Transformers for reasoning! Poster #272: Hall 3 + 2B. Sat 26th April, 10am - 12:30pm Singapore time

Excited to share our new method ✏️PENCIL! It decouples space complexity from time complexity in LLM reasoning, by allowing model to recursively erase and generate thoughts. Joint work w. my student Chenxiao Yang , along with Nati Srebro Bartom and David McAllester.