"Energy continuously flows from being concentrated, to becoming dispersed, spread out, wasted and useless." ⚡➡️🌬️ Sharing our work on the inability of softmax in Transformers to _robustly_ learn sharp functions out-of-distribution. Together w/ Christos Perivolaropoulos Federico Barbero & Razvan!

Round and Round we Go! 🔄 Rotary Positional Encodings (RoPE) are a common staple of frontier LLMs. _Why_ do they work so well, and _how_ do LLMs make advantage of them? The results might surprise you, as they challenge commonly-held wisdom! Read on ↩️ Work led by Federico Barbero!

We hope our work contributes to improved understanding of rotary PEs and how they're used, while paving the way to exciting positional embedding schemes in the future! Our work is available on the arXiv: arxiv.org/abs/2410.06205 Federico Barbero Christos Perivolaropoulos et al -- it's been a pleasure!

The cards will be revealed soon enough 👀

I love a good leaderboard... or several! ⏫ TGR 🐅 is our graph rewiring method for temporal graphs leveraging expander graph propagation. Turns out, TGR is _real good_ 🔥 setting SOTA on _four_ diverse tasks in the TGB dataset. Read on for more 🧵 Katarina Petrovic Shenyang Huang

Releasing my detailed commented introduction to LLM sampling colab.research.google.com/drive/18-2Z4TM… We get back to the basics and slowly build up to a reproduction of the adaptive temperature strategy from "Softmax is not enough" (from Petar Veličković et al.)

CGP has now been accepted to Learning on Graphs Conference 2025 😊🎉 Heartfelt congratulations to JJ Wilson on getting his first lead-author paper out of the door -- you've come such a long way from last year, in spite of having no formal AI training! I am impressed. 🙇‍♂️ Onwards! 🚀

Take time to study carefully how data flows in a Transformer, and you'll near-certainly find something useful that others missed 🧑‍🔬 Also, don't hold back on sharing a result just 'cause it feels obvious. 'softmax is not enough' was needlessly delayed for 10 months due to this 😶

Serious question to torch users who are to some degree aware of JAX and keras3: why do you prefer torch?

This is the poster I'm most happy about in my career, even though the actual amount of writing effort was minimal 🙃 Coming soon to NeurIPS Workshops near you (two spotlights)! 🔦 I sadly won't be there myself, but Christos Perivolaropoulos & Federico Barbero will be happy to tell you all about it 🚀

Our team is hiring Student Researchers Google DeepMind for '25! 🧑‍🔬 Interested in understanding reasoning capabilities from first principles? 🧑‍🎓 Currently studying for a BS/MS/PhD? 🧑‍💻 Have solid engineering and research skills? 🌟 We want to hear from you! Details in thread.

Heading tomorrow to Vancouver for NeurIPS! Please do reach out if you want to chat about reasoning in Transformers / LLMs :) I'll be presenting our work "Transformers need glasses! 👓" on Thursday at 4:30pm at East Exhibit Hall A-C #1806.

As I've been asked a few times recently: I won't be going to NeurIPS (travelled way too much! 😅) But if you're going, I warmly invite you to stop by Federico Barbero's poster this Thursday! Federico worked on this as part of his Student Researcher placement with us Google DeepMind

Thank you to the Scientific Methods for Understanding Deep Learning Workshop at #NeurIPS2024 for featuring our paper as one of the best paper runner-ups in its 'Debunking challenge'!!! 🚀🧑‍🔬 Also, sincere thanks to Christos Perivolaropoulos and Federico Barbero for their tireless work presenting our little softmax study throughout the day! 🙌

Kudos to everyone who worked on this. It looks amazing!

Making LLMs run efficiently can feel scary, but scaling isn’t magic, it’s math! We wanted to demystify the “systems view” of LLMs and wrote a little textbook called “How To Scale Your Model” which we’re releasing today. 1/n

I love the game of Boggle. This demo showcases our Gemini 2.0 Pro model’s coding abilities in AI Studio. It is mind boggling to think that it can write the full piece of code, including all the right data structures and search algorithms to find all valid words on a Boggle

Super excited to be heading to Singapore tomorrow to present our work on RoPE with Alex, Christos Perivolaropoulos, Razvan, Petar Veličković. Christos and I will be presenting on Fri 25 Apr 7 p.m. PDT — 9:30 p.m. PDT Hall 3 + Hall 2B #242. Happy to meet and catch up :) DMs are open!

Vol:18 No:5 → Dandelion: Smaller Clusters, Bigger Speeds—Distributed Transactions Redefined vldb.org/pvldb/vol18/p1…

AlphaEvolve is here! 🧬 this is one special system (especially when optimising things with jagged edges 😊) -- had a fantastic time using it! congrats Alexander Novikov Matej Balog Ngân Vũ (NV) and team!! 🚀 you can register your interest in using it through the link below: