You didn’t believe in Differential Private training for foundation models? We achieved the same performance as non-private MAE trained on the same dataset, but with rigorous DP. Code is released: github.com/facebookresear…. Presenting tomorrow at ICML, 11:30AM poster, #2313

We're looking for a postdoc at Meta FAIR to work on AI4Math, e.g., neural theorem proving, autoformalization, learning mathematical rules and abstractions, and automated discovery/conjecturing in math. Please apply at metacareers.com/jobs/145969190… and email me. Please help share this

github.com/facebookresear… by Guillaume Lample @ NeurIPS 2024 Timothee Lacroix Marie-Anne Lachaux Aurelien Rodriguez Amaury Hayat Thibaut Lavril Gabriel Ebner Xavier M and others!

LM Transparency Tool: Interactive Tool for Analyzing Transformer Language Models ➡️ go.fb.me/5ngnnp

🤩

Prof. Anima Anandkumar I really worry that the, shall we say, “questionable” claims in this paper (162 unproven by human theorems) will get taken seriously and the rest of us working in this field will look really bad for it. There are much better works already in this field.

New generation of cross-lingual embedding models -- the same could work for code + math

🚨 RELEASE ALERT ‼️ github.com/facebookresear… THIS CHANGES EVERYTHING $META just dropped a game-changing codebase! Now everyone can do LLM research! 😱 🧵10 best things people are already building with lingua 🔥👇

Want to do research in code generation with LLMs and wonky deep learning from the 90s? We're recruiting one Master student (M2) intern for 2025 at FAIR Paris in my team metacareers.com/jobs/106871446…

Meta Lingua: a minimal, fast LLM codebase for training and inference. By researchers, for researchers. Easily hackable, still reproducible. Built-in efficiency, profiling (cpu, gpu and mem) and interpretability (automatic activation and gradient statistics) Joint work w/ Badr Youbi Idrissi

The PyTorch reference transformer implementation for researchers. Amazing work!

🚨 New intern position in the FAIR CodeGen Team! 🚨 I'm particularly interested in working with candidates with expertise in off-policy RL methods, and/or in code generation with LLMs, but this is not a hard requirement and the project topic is somewhat flexible.

Why do we treat train and test times so differently? Why is one “training” and the other “in-context learning”? Just take a few gradients during test-time — a simple way to increase test time compute — and get a SoTA in ARC public validation set 61%=avg. human score! ARC Prize

Still a bit to go

We have just launched the new Lean reference manual, our core documentation intended as a comprehensive, precise description of Lean! #leanlang #leanprover Check out the manual: lean-lang.org/doc/reference/… Read more about the release: lean-lang.org/blog/2024-12-1…

Causal multi-token prediction at scale!

x.com/i/article/1904…

🚨 Your RL only improves 𝗽𝗮𝘀𝘀@𝟭, not 𝗽𝗮𝘀𝘀@𝗸? 🚨 That’s not a bug — it’s a 𝗳𝗲𝗮𝘁𝘂𝗿𝗲 𝗼𝗳 𝘁𝗵𝗲 𝗼𝗯𝗷𝗲𝗰𝘁𝗶𝘃𝗲 you’re optimizing. You get what you optimize for. If you want better pass@k, you need to optimize for pass@k at training time. 🧵 How?

We present an Autoregressive U-Net that incorporates tokenization inside the model, pooling raw bytes into words then word-groups. AU-Net focuses most of its compute on building latent vectors that correspond to larger units of meaning. Joint work with Badr Youbi Idrissi 1/8

🚀New paper alert! 🚀 In our work AI at Meta we dive into the struggles of mixing languages in largely multilingual Transformer encoders and use the analysis as a tool to better design multilingual models to obtain optimal performance. 📄: arxiv.org/abs/2508.02256 🧵(1/n)