GPTs are generating sequences in a left-to-right order. Is there another way? With François Fleuret and @evanncourdier, in partnership with SkySoft-ATM, we developed σ-GPT, capable of generating sequences in any order chosen dynamically at inference time. 1/6

Enjoy attention? Want to make it ~18% faster? Try out Sigmoid Attention. We replace the traditional softmax in attention with a sigmoid and a constant (not learned) scalar bias based on the sequence length. Paper: arxiv.org/abs/2409.04431 Code: github.com/apple/ml-sigmo… This was

Excited to present "Recurrent Complex-Weighted Autoencoders for Unsupervised Object Discovery" at #NeurIPS2024! TL;DR: Our model, SynCx, greatly simplifies the inductive biases and training procedures of current state-of-the-art synchrony models. Thread 👇 1/x.

📢 New Paper: Thinking Slow, Fast: Scaling Inference Compute with Distilled Reasoners. w/ Junxiong Wang, Matteo Pagliardini, Kevin Li , Aviv Bick, Zico Kolter, Albert Gu, François Fleuret , Tri Dao. Paper: arxiv.org/abs/2502.20339 pic.x.com/NyIzBefWcR 1/N

🚀 Together with ETH Zürich and the CSCS, we have just released Apertus, 🇨🇭 Switzerland’s first large-scale, open, multilingual language model — a milestone in generative AI for transparency and diversity. Find out more: go.epfl.ch/a672aa

The next generation of open LLMs should be inclusive, compliant, and multilingual by design. That’s why we (EPFL ETH Zurich) built Apertus.

New work from our MLO lab EPFL: Benchmarking the variety of different proposed LLM optimizers: Muon, AdEMAMix, ... all in the same setting, tuned, with varying model size, batch size, and training duration! Huge sweep of experiments by Andrei Semenov Matteo Pagliardini M Jaggi

Amazing "competing" work from Kaiyue Wen Tengyu Ma Percy Liang There are some good stories about optimizers to tell this week 😃 arxiv.org/abs/2509.01440 arxiv.org/abs/2509.02046