🔥The 9th Round of Easy Loan, Earn $40 Reward is in progress❗️ ⏰ Promotion Period: January 15th - Feburary 15th, 2025 👉 Register now and check more details at gate.io/campaigns/358

In ML, we train on biased (huge) datasets ➡️ models encode spurious corrs and fail on minority groups. Can we scalably remove "bad" data? w/ Saachi Jain Kimia Hamidieh Kristian Georgiev Andrew Ilyas Marzyeh we propose D3M, a method for exactly this: gradientscience.org/d3m/

Traditional concept vectors used to explain deep representations fail to compose when combined, i.e. 🐤(small) +🦢(white) =🦩(big & colorful)❌ We propose CCE: a method for extracting *composable* concepts, i.e. 🐤(small) +🦢(white) =🕊️(small & white)✅ debugml.github.io/compositional-…

Excited to share something I was working on for a while now: the Before AGI podcast! There is a lot of talking about AGI. But what’s less explored is an equally important question: What happens before AGI arrives? And what should we be doing to prepare? First episode with

At #ICML2024 ? Our tutorial "Data Attribution at Scale" will be to tomorrow at 9:30 AM CEST in Hall A1! I will not be able to make it (but will arrive later that day), but my awesome students Andrew Ilyas Sam Park Logan Engstrom will carry the torch :)

Attending #ICML2024? Check out our work on decomposing predictions and editing model behavior via targeted interventions to model internals! Poster: #2513, Hall C 4-9, 1:30p (Tue) Paper: arxiv.org/abs/2404.11534 w/ Harshay Shah Andrew Ilyas

On Saturday, I will be giving a talk at 11:40 am at the ICML @agenticmarkets workshop on our work on three-layer data markets: arxiv.org/abs/2402.09697. I would discuss a model to study data monetization and the role of privacy regulations in the presence of strategic agents!

Thanks to all who attended our tutorial "Data Attribution at Scale" at ICML (w/ Sam Park Logan Engstrom Kristian Georgiev Aleksander Madry)! We're really excited to see the response to this emerging topic. Slides, notes, ICML video: ml-data-tutorial.org Public recording soon!

The ATTRIB workshop is back @ NeurIPS 2024! We welcome papers connecting model behavior to data, algorithms, parameters, scale, or anything else. Submit by Sep 18! More info: attrib-workshop.cc Co-organizers: Tolga Bolukbasi Logan Engstrom Sadhika Malladi Elisa Nguyen Sam Park

ContextCite Attributing Model Generation to Context paper page: huggingface.co/papers/2409.00… How do language models use information provided as context when generating a response? Can we infer whether a particular generated statement is actually grounded in the context, a

Machine unlearning ("removing" training data from a trained ML model) is a hard, important problem. Datamodel Matching (DMM): a new unlearning paradigm with strong empirical performance! w/ Kristian Georgiev Roy Rinberg Sam Park Shivam Garg Aleksander Madry Seth Neel (1/4)

DMM is a *meta-algorithm*, so better data attribution ➡️ better oracle predictions ➡️ better unlearning! Check out our work for details on DMM, new techniques for evaluating unlearning, theoretical analyses, and more! arXiv: arxiv.org/abs/2410.23232 Blog: bit.ly/unlearning-via…

I wrote a survey article on computationally efficient methods for "robust" mean estimation, which includes robustness to contamination, heavy-tailed data, or in the sense of differential privacy. The same ideas are useful for all 3 (seemingly-different) forms of robustness! 1/2

Excited to see this new paper published in Transactions on Machine Learning Research! I study the problem of simultaneously estimating many private regressions that share the same set of covariates X but have l different outcomes Y. For example, X might be a persons genomic data, and Y's might correspond to

Andrej Karpathy The hottest new programming language is vibes

Clean mathematical explanation for why critical windows appear for any localization-based sampler! Also interesting that in LLMs, these often correspond to failures in reasoning. Congrats Marvin Li and Aayush Karan for the cool mix of theory and empirical work!

le chat doesn’t disappoint

large reasoning models are extremely good at reward hacking. A thread of examples from OpenAI's recent monitoring paper: (0/n)

Want state-of-the-art data curation, data poisoning & more? Just do gradient descent! w/ Andrew Ilyas Ben Chen Axel Feldmann Billy Moses Aleksander Madry: we show how to optimize final model loss wrt any continuous variable. Key idea: Metagradients (grads through model training)

We’ve been hard at work on reinforcement fine-tuning (RFT) to make it a more flexible and powerful tool. RFT is best thought of as a way to improve model capabilities on well-specified tasks with known correct answers. It shouldn't come as a surprise that models can often get

Why do larger language models generalize better? In our new ICLR paper, we derive an interpretable generalization bound showing that compute-optimal LLMs provably generalize better with scale! 📄arxiv.org/abs/2504.15208 1/7🧵