The terms “CoT” and reasoning trace make it sound like the CoT is a summary of an LLM’s reasoning. But IMO it’s more accurate to view CoT as a tool models use to think better. CoT monitoring is about tracking how models use this tool so we can glean insight into their

1/ In my job market paper (with Luca Henkel) in partnership with PSG Global Solutions, TP subsidiary, we tested AI-led interviews with 70,000 applicants.

we're going to research the impact of AI on totally greenfield software engineering. come be in our anonymized data/have fun/maybe win $15k, in SF, september 6! x.com/FactoryAI/stat…

thrilled to be on Patrick McKenzie's complex systems podcast! we talk about perception vs reality in the impact of AI on dev productivity, the industrial organization of software engineering, and much more complexsystemspodcast.com/episodes/the-g…

> 🤖 Half of participants will build with AI tools > 👩‍💻 Half of participants will build without AI tools > Judging is blind and scored on speed, precision, and quality.

This week on Complex Systems I was thrilled to welcome METR's Joel Becker to chat about how we rigorously measure progress of LLMs, an interesting research result Joel and the team hit recently, and a tiny bit about the industrial organization of software engineering.