🌎Introducing LINGOLY, our new reasoning benchmark that stumps even top LLMs (best models only reach ~35% accuracy)🥴 In a colab between University of Oxford, Stanford University and UK Linguistic Olympiad puzzle authors, we stress test LLMs on over 90 low-resource and extinct languages...

Big congratulations to my DPhil student Andrew Bean on an excellent presentation on benchmarking Olympiad-level linguistic reasoning puzzles in low-resource languages at the Meta Open Innovation Research Community event in London (Oct 29, 2024)! 🎉

Super excited to see PRISM recognised as a #NeurIPS2024 best paper. This was an incredible large-scale effort by Hannah Rose Kirk and fantastic collaborators. If you're interested in human feedback, check it out, there are 100+ pages of detailed insights! 🔥

A real honour and career dream that PRISM has won a NeurIPS Conference best paper award! 🌈 One year ago I was sat in a 13,000+ person audience of NeurIPs '23 having just finished data collection. Safe to say I've gone from feeling #stressed to very #blessed 😁