🔥The 9th Round of Easy Loan, Earn $40 Reward is in progress❗️ ⏰ Promotion Period: January 15th - Feburary 15th, 2025 👉 Register now and check more details at gate.io/campaigns/358

🎉 Two of my papers have been accepted this week at #ICLR2024 & #AISTATS! Big thanks and congrats to co-authors Xuxi Chen & Eric Gan, mentors Atlas Wang & Gintare Karolina Dziugaite, and especially my advisor Baharan Mirzasoleiman! 🙏 More details on both papers after the ICML deadline!

We've finally put out a detailed IEEE/ACM paper on Google's multi-year effort to ease the burden of code review with ML. Google engineers now resolve 7.5% of all code review comments with an ML-suggested edit. But the path to that number has been a fun ML and UX journey!

📢Mixtures of Experts unlock parameter scaling for deep RL! Adding MoEs, and in particular Soft MoEs, to value-based deep RL agents results in more parameter-scalable models. Performance keeps increasing as we increase number of experts (green line below)! 1/9

In deep nets, we observe good generalization together with memorization. In this new work, we show that, in stochastic convex optimization, memorization of most of the training data is a necessary feature of optimal learning.

DEADLINE March 29: prepare and submit your application for EEML 2024, Novi Sad, Serbia eeml.eu 🇷🇸. Topics: Basics of ML, Multimodal learning, NLP, Advanced DL architectures, Generative models, AI for Science. Check our stellar speakers! Scholarships available! 🎉

Google presents Mixture-of-Depths Dynamically allocating compute in transformer-based language models Same performance w/ a fraction of the FLOPs per forward pass arxiv.org/abs/2404.02258

How are LLM capabilities affected by pruning? Checkout our ICLR 2026 paper showing that ICL is preserved until high levels of sparsity, in contrast to fact recall which quickly deteriorates. Our analysis reveals which part of the network is more prunable for a given capability.

See u tmrw ICLR 2025 Sess 1 #133! When we down-scale LLMs (e.g.pruning), what happens to their capabilities? We studied complementary skills of memory recall & in-context learning and consistently found that memory recall deteriorates much quicker than ICL when down-scaling.

thrilled this paper was accepted to #ICML2024 , looking forward to chatting about this with many of you in vienna! camera-ready version available at arxiv.org/abs/2402.08609 w/ Johan S. Obando 👍🏽 Ghada Sokar Timon Willi Clare Lyle Jesse Farebrother Jakob Foerster Gintare Karolina Dziugaite & doina precup

We've seen memorization in NNets, despite good generalization. But can we generalize without memorizing? Come hear about our best #ICML2024 paper award work on showing that in stochastic convex opt, optimal learners memorize! Talk today at 11.15am, poster at 11.30am.

Excited to present our spotlight paper on MoEs in RL today at #ICML2024! Me, Johan S. Obando 👍🏽, Pablo Samuel Castro, and Jesse Farebrother are looking forward to chat with you! Poster #1207 Hall C 4-9 at 1:30-3:00 pm

#ECMLPKDD 2024 is happening in my beautiful hometown Vilnius this week! There? Come see my keynote on memorization and generalization this evening @ 6pm.

Excited to share our new mechanistic unlearning method that targets the mechanisms behind factual recall for more robust and effective knowledge removal, making relearning more difficult 💪

We are hiring Applied Interpretability researchers on the GDM Mech Interp Team!🧵 If interpretability is ever going to be useful, we need it to be applied at the frontier. Come work with Neel Nanda, the Google DeepMind AGI Safety team, and me: apply by 28th February as a

Introducing Learned Asynchronous Decoding w/ friends from MIT/Google! LLM responses often have chunks of tokens that are semantically independent. We train LLMs to identify and decode them in parallel, speeding up inference by 1.46x geomean (AlpacaEval) w/ only 1.3% quality loss.

Come chat with us @ ICLR on April 24th poster session to learn more about what matters in sparse LLM pretraining!

I will travelling to Singapore 🇸🇬 this week for the ICLR 2025 Workshop on Sparsity in LLMs (SLLM) that I'm co-organizing! We have an exciting lineup of invited speakers and panelists including Dan Alistarh, Gintare Karolina Dziugaite, Pavlo Molchanov, Vithu Thangarasa, Yuandong Tian and Amir Yazdan.

Managed to get through the ICLR registration line? Come to our poster to learn about applying domain adversarial training for unlearning. Starting now @ Hall 3 #506

Sparse LLM workshop will run on Sunday with two poster sessions, a mentoring session, 4 spotlight talks, 4 invited talks and a panel session. We'll host an amazing lineup of researchers: Dan Alistarh Vithu Thangarasa Yuandong Tian Amir Yazdan Gintare Karolina Dziugaite Olivia Hsu Pavlo Molchanov Yang Yu

1/10 🧵 🔍Can weight symmetry provide insights into sparse training and the Lottery Ticket Hypothesis? 🧐We dive deep into this question in our latest paper, "Sparse Training from Random Initialization: Aligning Lottery Ticket Masks using Weight Symmetry", accepted at #ICML2025