🔥The 9th Round of Easy Loan, Earn $40 Reward is in progress❗️ ⏰ Promotion Period: January 15th - Feburary 15th, 2025 👉 Register now and check more details at gate.io/campaigns/358

It's very difficult to improve the *exponent* in scaling laws for loss vs compute, especially by changing the optimizer! Our new paper shows that scaling momentum correctly can *provably* improve the scaling exponent on a theoretical model. Empirically, it works on LSTMs too!

join us tonight to talk about Adam! maybe we will touch a bit on Muon & friends -- they carry many of the open questions we have about Adam ❤️ thanks Yannic

😻

Best reference for understanding and improving new optimizers!

Thanks Robert for this amazing summary!!! ❤️

Forecasting future events is a fascinating task for language models. Arguably the hardest application for a pure "oracle" that can't take actions; requiring reasoning about conflicting info, planning, information seeking... But, forecasting is also uniquely hard to evaluate:

If you are ever ablating on LM pretraining, this is the ONLY codebase I trust, by the amazing Nico.

It is essential to thoroughly evaluate, test, and compare ideas. This unbiased process is rare in modern research. Niccolò did this for averaging checkpoints: with a large number of experiments, demonstrating when and where to average weights in modern large-scale setups. Super

Text-to-LoRA: What if you no longer had to fine-tune your LLM for every single downstream task? 🚀 Stoked to share our work on instant LLM adaptation using meta-learned hypernetworks 📝 → 🔥 The idea is simple yet elegant: We text-condition a hypernetwork to output LoRA

*Generalized Interpolating Discrete Diffusion* by Dimitri von Rütte Antonio Orvieto & al. A class of discrete diffusion models combining standard masking with uniform noise to allow the model to potentially "correct" previously wrong tokens. arxiv.org/abs/2503.04482

cool work on linear rnn theory!!