Our interpretability team recently released research that traced the thoughts of a large language model. Now we’re open-sourcing the method. Researchers can generate “attribution graphs” like those in our study, and explore them interactively.

Feel like I haven't seen much work on incorporating existing models into VLMs. Seems a bit naive, but I'd imagine incorporating embeddings from depth estimation / segmentation / edge detection models would help remedy the "text first, vision second" issue present in VLMs

🔠 UTF-8 was never meant for language models. Yet every major tokenizer still uses it, creating unfair "byte premiums". Why should your native script cost more to tokenize? It's time for a change. 🧵👇

Really cool work - as a field, we all need more ethically sourced and openly licensed data

⏰ We introduce Reinforcement Pre-Training (RPT🍒) — reframing next-token prediction as a reasoning task using RLVR ✅ General-purpose reasoning 📑 Scalable RL on web corpus 📈 Stronger pre-training + RLVR results 🚀 Allow allocate more compute on specific tokens

After the o3 price reduction, we retested the o3-2025-04-16 model on ARC-AGI to determine whether its performance had changed. We compared the retest results with the original results and observed no difference in performance.

rip gcp / aistudio, how am i supposed to get my work done without system prompt control

Our computer vision textbook is now available for free online here: visionbook.mit.edu We are working on adding some interactive components like search and (beta) integration with LLMs. Hope this is useful and feel free to submit Github issues to help us improve the text!

Really like the approach of treating multitude training as a series of database transactions+rollbacks. Makes intuitive sense, surprised I haven’t seen this discussed elsewhere in the OSS ML space as of yet

In the beginning, there was BERT. Eventually BERT gave rise to RoBERTa. Then, DeBERTa. Later, ModernBERT. And now, NeoBERT. The new state-of-the-art small-sized encoder:

OCR is not yet solved

For anyone serious about on-device AI: more <0.5B LLMs are desperately needed

Marin is such a cool project

👀

Providing a YouTube URL to Google AI Studio w/ Gemini 2.5 Pro, seeing significant spikes in GPU utilization while I wait on a model response (usage drop is when the window is no longer active). Anyone familiar with this? Unsure why any of this would be client-side.

uh oh

New paper & surprising result. LLMs transmit traits to other models via hidden signals in data. Datasets consisting only of 3-digit numbers can transmit a love for owls, or evil tendencies. 🧵