🔥The 9th Round of Easy Loan, Earn $40 Reward is in progress❗️ ⏰ Promotion Period: January 15th - Feburary 15th, 2025 👉 Register now and check more details at gate.io/campaigns/358

Fixed the missing newline at the end of the file (nothing to see here) (move along)

When will AI systems be able to carry out long projects independently? In new research, we find a kind of “Moore’s Law for AI agents”: the length of tasks that AIs can do is doubling about every 7 months.

🧬 Meet Lyra, a new paradigm for accessible, powerful modeling of biological sequences. Lyra is a lightweight SSM achieving SOTA performance across DNA, RNA, and protein tasks—yet up to 120,000x smaller than foundation models (ESM, Evo). Bonus: you can train it on your Mac. read

Having worked a lot on evaluating agents, manually reading through actual transcripts is core to understanding bottlenecks & finding bugs. I've found that Docent makes this much easier!

We're entering an era where AI outputs are becoming so vast, humans alone can't analyze them. Today's LLMs produce tens of thousands of tokens per task—but complex challenges like comprehensive cancer research, inventing novel molecules, or building entire codebases will soon

Contamination has been a concern for ~every eval I've worked on, but is not easy to get quantitative signals on. See Kevin Meng's thread on finding subtle, *qualitative* evidence of contamination using Docent!

GPT-4o image gen is pretty sick but its clocks are still stuck at 10:10

We’re releasing PaperBench, a benchmark evaluating the ability of AI agents to replicate state-of-the-art AI research, as part of our Preparedness Framework. Agents must replicate top ICML 2024 papers, including understanding the paper, writing code, and executing experiments.

We tested a pre-release version of o3 and found that it frequently fabricates actions it never took, and then elaborately justifies these actions when confronted. We were surprised, so we dug deeper 🔎🧵(1/) x.com/OpenAI/status/…

Pretty striking follow-up finding from our o3 investigations: in the chain of thought summary, o3 plans to tell the truth — but then it makes something up anyway!

Let's chat 👇

Our MLE-bench poster #367 is up till 12:30pm in Hall 3, and our oral presentation is at 3:30pm today in Garnet 213-215. Come say hi!

The ChatGPT sycophancy thing shows that RLHF is hard and it's challenges aren't going away any time soon. It's required to make these models we love. It's being ignored with the other hype around RL. The RLVR stuff will "saturate" like pretraining and RLHF is never fully solved.