🔥The 9th Round of Easy Loan, Earn $40 Reward is in progress❗️ ⏰ Promotion Period: January 15th - Feburary 15th, 2025 👉 Register now and check more details at gate.io/campaigns/358

Neural nets can generalize well on test data, but often lack robustness to distributional shifts & adversarial attacks. Our #NeurIPS2020 paper on simplicity bias sheds light on this phenomenon. Poster: session #4, town A2, spot C0, 12pm ET today! Paper: bit.ly/39RXDel

Do input gradients highlight discriminative and task-relevant features? Our #NeurIPS2021 paper takes a three-pronged approach to evaluate the fidelity of input gradient attributions. Poster: session 3, spot C0 Paper: bit.ly/3EzdvyH with Prateek Jain and @pnetrapalli

You’re deploying an ML system, choosing between two models trained w/ diff algs. Same training data, same acc... how do you differentiate their behavior? ModelDiff (gradientscience.org/modeldiff) lets you compare *any* two learning algs! w/ Harshay Shah Sam Park Andrew Ilyas (1/8)

TRAK, our latest work on data attribution (trak.csail.mit.edu), speeds up datamodels up to 1000x! ➡️ our earlier work ModelDiff (w/ Harshay Shah Sam Park Aleksander Madry) can now compare any two learning algorithms in larger-scale settings. Try it out: github.com/MadryLab/model…

If you are at #ICML2023 today, check out our work on ModelDiff, a model-agnostic framework for pinpointing differences between any two (supervised) learning algorithms! Poster: #407 at 2pm (Wednesday) Paper: icml.cc/virtual/2023/p… w/ Sam Park Andrew Ilyas Aleksander Madry

New work with Andrew Ilyas and Aleksander Madry on tracing predictions back to individual components (conv filters, attn heads) in the model! Paper: arxiv.org/abs/2404.11534 Thread: 👇

How is an LLM actually using the info given to it in its context? Is it misinterpreting anything or making things up? Introducing ContextCite: a simple method for attributing LLM responses back to the context: gradientscience.org/contextcite w/ Ben Cohen-Wang, Harshay Shah, Kristian Georgiev

♥️

How do black-box neural networks transform raw data into predictions? Inside these models are thousands of simple "components" working together. New MIT CSAIL research (bit.ly/473lcfE) introduces a method that helps us understand how these components compose to affect

How can we really know if a chatbot is giving a reliable answer? 🧵 MIT CSAIL’s "ContextCite" tool can ID the parts of external context used to generate any particular statement from a language model, improving trust by helping users easily verify the statement:

Wake up babe New MoE scaling laws dropped

MoEs provide two knobs for scaling: model size (total params) + FLOPs-per-token (via active params). What’s the right scaling strategy? And how does it depend on the pretraining budget? Our work introduces sparsity-aware scaling laws for MoE LMs to tackle these questions! 🧵👇

It can be helpful to pinpoint the in-context information that a language model uses when generating content (is it using provided documents? or its own intermediate thoughts?). We present Attribution with Attention (AT2), a method for doing so efficiently and reliably! (1/8)

If you’re at #ICLR2025, go watch Vimal Thilak🦉🐒 give an oral presentation at the @SparseLLMs workshop on scaling laws for pretraining MoE LMs! Had a great time co-leading this project with Samira Abnar & Vimal Thilak🦉🐒 at Apple MLR last summer. When: Sun Apr 27, 9:30a Where: Hall 4-07