🔥The 9th Round of Easy Loan, Earn $40 Reward is in progress❗️ ⏰ Promotion Period: January 15th - Feburary 15th, 2025 👉 Register now and check more details at gate.io/campaigns/358

In our test, llama3-instruct got insane scores on IFEval (8b: 74.1, 70b: 83.2, GPT-4: 79.7) which mean excellent instruction-following. 😯 However on MATH, we asked models to put the answer in \boxed{} and got some wired output 🤔 What would be the reason?

Wanna train PRMs but process labels, annotated manually or automatically, sound too expensive to you😖? Introduce Implicit PRM🚀 – Get your model free process rewards by training an ORM on the cheaper response-level data, with a simple parameterization at no additional cost💰!

How to unlock advanced reasoning via scalable RL? 🚀Introducing PRIME (Process Reinforcement through Implicit Rewards) and Eurus-2, trained from Base model to surpass Qwen2.5-Math-Instruct using only 1/10 of the data. We're still scaling up - w/ 3x more training data to go! 🧵

Kyle Corbitt Nathan Lambert That’s exactly our motivation! An offline RM may suffer from reward hacking, so we online update it with latest policy rollouts and ground truth labels in a scalable way. This is the key of PRIME. gt reward alone can go a long way, but online rm pushes the limits further.

The DeepSeek bros share the same view with us on PRM: If it can't **scale** with online updates, it has no place in RL. Implicit PRM & PRIME are exactly born for this!

PRIME adopted by Prime!

Process Reinforcement through Implicit Rewards

1/ PRIME is alive on arXiv💡! Building on our blog, we've added extensive experiments exploring: - Implicit PRM design choices - PRIME's integration with other RL algorithms - Value models vs. PRMs - RL from base models (“Zero”) See details below🧵 arxiv.org/abs/2502.01456

Really curious how they use PRIME🤔

A nice insight on scaling RL -- we gotta let the models keep entropy up.