It was great to be able to present my glitch token work at AI Mad Lab in Oslo. As always, great community, excellent conversations, looking forward to many more meetups.

Quasar Alpha uses cl200k, like gpt4o. ម្បី means what for max glitchiness and occasional errors. Also seems to induce some reasoning-like traces at times.

📝 Submit papers (up to 9 pages, shorter submission ) via OpenReview: openreview.net/group?id=ICML.… 🗓️ Important dates: Deadline: May 30, 2025 Notifications: June 9, 2025 Workshop: July 18, 2025 Both archival and non-archival options available! #ICML2025 #TokShop #ML #NLProc

Did you know BPE (Byte Pair Encoding), the most common LLM tokenizer, was originally a compression algorithm from 1994? #Tokenization #LLM #NLP Want to find out more about tokenization? Join our workshop at ICML! tokenization-workshop.github.io

Sander and I have been working on a new encoding scheme for tokenization which mitigates variable length byte sequences for different scripts, prevents partial UTF-8 byte tokens, and offers a simple and efficient pretokenization alternative to regular expressions!

In other words: (graphic design by Sander Land)

I’m thrilled to share RewardBench 2 📊— We created a new multi-domain reward model evaluation that is substantially harder than RewardBench, we trained and released 70 reward models, and we gained insights about reward modeling benchmarks and downstream performance!

Thank you to co-authors Nathan Lambert, Valentina Pyatkin, Sander Land, Jacob Morrison, Noah A. Smith, and Hanna Hajishirzi for a great collaboration! Read more in the paper here (ArXiv soon!): github.com/allenai/reward… Dataset, leaderboard, and models here: huggingface.co/collections/al…

Huge congrats to all the authors Diana Abagyan, Alejandro, Felipe Cruz-Salinas, Kris Cao, John Lin, Acyr Locatelli, Marzieh Fadaee, Ahmet Üstün. I always enjoy collabs which tackle learning efficiency as an explicit design choice — rather than post training fixes. arxiv.org/abs/2506.10766

We are launching a new speaker series at EleutherAI, focused on promoting recent research by our team and community members. Our first talk is by Catherine Arnett on tokenizers, their limitations, and how to improve them.