🔥The 9th Round of Easy Loan, Earn $40 Reward is in progress❗️ ⏰ Promotion Period: January 15th - Feburary 15th, 2025 👉 Register now and check more details at gate.io/campaigns/358

where were you when GPQA was killed

one must imagine paris hilton getting 80% MFU

I will be at PyTorch Conf all tomorrow! get in touch if you wanna say hi, or swing by the panel I’m on @ 4:45 !

'opportunities and risks' explosion by 2027

Highly recommend this resource on evals from Clémentine Fourrier 🍊 !!

unrelated to tonight, made hailey , mutuals follow for more hailey Content if you’re into that sorta thing

Excited to release FrontierMath, an exceptionally challenging benchmark of original expert-level mathematics problems written by 60 leading mathematicians. Current top models solve less than 2% correctly.

Epoch continues to be on a roll! incredible work as always—benchmarks are the foundation for further improvements and we’re long overdue for harder ones!

Excited for our Benchmarking Hub: independent evals to build shared understanding of AI capabilities. Coming soon: • More benchmarks: FrontierMath, SWE-Bench (basically all the best ones) • Predictive work—aiming to be for AI what FiveThirtyEight is for elections

Check out our NeurIPS tutorial on meta-generation methods like majority voting and best-of-N, presented with the awesome Sean Welleck Matthew Finlayson ✈️ NeurIPS Ilia Kulikov ! I’ll be presenting this and other work I did at EleutherAI next week at NeurIPS, come say hi!

1/9 We’re announcing the development of Tier 4, a new suite of math problems that go beyond the hardest problems in FrontierMath. o3’s performance is remarkable, but there’s still a ways to go before any single AI system nears the collective genius of the math community.

from the field that brought you ‘Laughing Hyena Distillery’

the only eval i trust now, and the vibes are immaculate

Claude Plays Pokémon continues on as a researcher's personal project. Follow along on Twitch: twitch.tv/claudeplayspok…

2 misconceptions from yesterday: - no training on playing videogames. Claude can just do this now - “why isn’t this a demo”

Excited to see how people will use the model and what engineers will build with it! Feeling privileged to have gotten the opportunity to work on it with an amazing team. If you’re interested in working on the next one, apply online - my team and many others are hiring!

opus 4 is here—“o4”, if you will