Where does one language model outperform the other? We examine this from first principles, performing unsupervised discovery of "abilities" that one model has and the other does not. Results show interesting differences between model classes, sizes and pre-/post-training.

There are still posts about 'new papers showing AI models cannot reason'. There are unfortunately problems into how these evaluations were done and also many of those limitations are known, peer-reviewed and published. Here is a simplified version of what's going on as far as I

That’s the vision of commit0: github.com/commit-0/commi… there is nearly zero improvement on this benchmark in the past few months. I don’t think this problem is solvable in 24 months…

The more I dive into LM training, the more I feel pretraining is just starting. Some questions I’m particularly interested in: * what data unlocks what capabilities? * do we train on capabilities sequentially or in parallel? * how many synthetic examples is a human example worth?

It's time to think about code generation beyond functional correctness. Refactoring multiple libraries requires designing APIs that support past and future use cases, which is challenging for even human engineers. Can't wait for LLMs to unify pytorch, tensorflow, and jax 😬

LM training bottlenecks 2024: code RL -> code execution is slower than model inference 2025: reasoning model RL -> rolling out 32k tokens takes forever maybe diffusion models are indeed the solution lol

✨Release: We upgraded SkyRL into a highly-modular, performant RL framework for training LLMs. We prioritized modularity—easily prototype new algorithms, environments, and training logic with minimal overhead. 🧵👇 Blog: novasky-ai.notion.site/skyrl-v01 Code: github.com/NovaSky-AI/Sky…

Dang, truly impressed by how an academic lab just figured out a lot of mysteries in mid-training to close the RL gap between llama and qwen: * length scheduler plays a key role to stabilize RL * there is some dark magic in prompt template? * the data interaction stuff is really

Congrats to team! They built my dream benchmark.

We don’t have AI self-improves yet, and when we do it will be a game-changer. With more wisdom now compared to the GPT-4 days, it's obvious that it will not be a “fast takeoff”, but rather extremely gradual across many years, probably a decade. The first thing to know is that

📢 today's scaling laws often don't work for predicting downstream task performance. For some pretraining setups, smooth and predictable scaling is the exception, not the rule. a quick read about scaling law fails: 📜arxiv.org/abs/2507.00885 🧵1/5👇

Do language models have algorithmic creativity? To find out, we built AlgoTune, a benchmark challenging agents to optimize 100+ algorithms like gzip compression, AES encryption and PCA. Frontier models struggle, finding only surface-level wins. Lots of headroom here!🧵⬇️

AI Research Agents are becoming proficient at machine learning tasks, but how can we help them search the space of candidate solutions and codebases? Read our new paper looking at MLE-Bench: arxiv.org/pdf/2507.02554 #LLM #Agents #MLEBench

haven't made a new blog post in over a year, so here's a new one: justintchiu.com/blog/sftrl/ it's short

I'll be around the ICML venue this afternoon. Message me if you want to meet! These days, I think about reasoning and RL. Also happy to talk about academia vs. industry (I think the lack of compute in academia is a feature not a bug), faculty and PhD student recruiting at UMass.

i've been thinking lately about how future ai systems will interact with us and how we can make systems that care about people and wanted to put words to it -- hopefully it resonates a bit!

🚀New dataset release: WildChat-4.8M 4.8M real user-ChatGPT conversations collected from our public chatbots: - 122K from reasoning models (o1-preview, o1-mini): represent real uses in the wild and very costly to collect - 2.5M from GPT-4o 🔗 hf.co/datasets/allen… (1/4)

wow, it's absolutely amazing to see in a more contamination-free setting, qwen is even better than claude. big W for open-source models too