🔥The 9th Round of Easy Loan, Earn $40 Reward is in progress❗️ ⏰ Promotion Period: January 15th - Feburary 15th, 2025 👉 Register now and check more details at gate.io/campaigns/358

📘 Could ChatGPT get an engineering degree? Spoiler, yes! In our new PNASNews article, we explore how AI assistants like GPT-4 perform in STEM university courses — and on average they pass a staggering 91.7% of core courses. 🧵 #AI #HigherEd #STEM #LLMs #NLProc

AI amplifying biorisk has been a major topic in policy & governance work 🦠 ⚠️ Does the available evidence match this level of attention? 🔬 We review the evidence to-date to provide grounded recommendations.

Introducing Global-MMLU🌍: A multilingual benchmark featuring MMLU translations in 42 languages crafted with: ✅ Human curation ✅ Extensive metadata ✅ Insights into cultural sensitivity Proud to have collaborated with Shivalika Singh and Cohere For AI to bring this work to life!

2. 🤝INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledge by Angelika Romanou, Negar Foroutan @ACL ✈️, Anna Sotnikova, Zeming Chen, harsha, Shivalika Singh, Rishabh Maheshwary, Micol Altomare, Mohamed A. Haggag, Imanol Schlag, Marzieh Fadaee, Sara Hooker, Antoine Bosselut

🟢LingOly: A Benchmark of Olympiad-Level Linguistic Reasoning Puzzles in Low-Resource and Extinct Languages Oral Session 4A, Thursday 15:30-16:30 TLDR: We construct a hard reasoning eval that controls for memorisation using low-resource languages. Work led by Andrew Bean

🚀 We've released the Lite version of Global-MMLU! ✅ Faster, easier evaluation ✅ Balanced culturally sensitive and culturally agnostic knowledge Try it at 🤗: huggingface.co/datasets/Coher…

Excited to share that we’ve been awarded an Evaluation Research grant from AI at Meta for our Multilingual Evaluation project! 🚀 This will help us advance research on assessing LLMs’ multilingual capabilities on regional knowledge. Many thanks to Meta for the support! 🙌

Our regional knowledge benchmark has been accepted at #ICLR2025 🌏🚀 I’m happy to have represented Philippines 🇵🇭 in this massive community effort led by Cohere for AI (Micol Altomare Shivalika Singh Sara Hooker et al) and EPFL (Angelika Romanou Negar Foroutan @ACL ✈️ Antoine Bosselut). You guys are

Thrilled to see INCLUDE accepted as a Spotlight at ICLR 2025! 🎉 This was a massive open science effort! Amazing work led by Angelika Romanou Negar Foroutan, Anna ❤️ Was lovely collaborating with them as well as harsha Rishabh Maheshwary and others from Cohere For AI community! 🙌

Excited to share that our paper, 'The LLM Language Network,' has been accepted to NAACL 2025! Looking forward to presenting it in Albuquerque—see you there! 🏜️ #NAACL2025

Are LLMs linguistically productive and systematic in morphologically-rich languages as good as humans? No 🤨 Our new NAACL 2025 paper (arxiv.org/abs/2410.12656) reveals a significant performance gap between LLMs and humans in linguistic creativity and morphological generalization.

We updated our Global-MMLU paper to describe our Global-MMLU lite ✨ eval set. This is a quicker more effficient eval on a subset of languages balanced with equal number of Culturally Sensitive (CS) and culturally Agnostic (CA) per language. We also added to eval harness 🔥

🚨 New Preprint!! LLMs trained on next-word prediction (NWP) show high alignment with brain recordings. But what drives this alignment—linguistic structure or world knowledge? And how does this alignment evolve during training? Our new paper explores these questions. 👇🧵

I am recruiting 2 PhD students for Fall'25 Computer Science at Aarhus University to work on bleeding-edge topics in #NLProc #LLMs #AIAgents (e.g. LLM reasoning, knowledge-seeking agents, and more). Details: cs.au.dk/~clan/openings Deadline: May 1, 2025 Please boost! cc: WikiResearch Pioneer Centre for AI SODAS, Copenhagen (Bluesky: @cphsodas.bsky.social)

NEW PAPER ALERT: Generating visual narratives to illustrate textual stories remains an open challenge, due to the lack of knowledge to constrain faithful and self-consistent generations. Our #CVPR2025 paper proposes a new benchmark, VinaBench, to address this challenge.

If you’re at ICLR 2026 this week, come check out our spotlight poster INCLUDE during the Thursday 3:00–5:30pm session! I will be there to chat about all things multilingual & multicultural evaluation. Feel free to reach out anytime during the conference. I’d love to connect!

Excited to be at #NAACL2025 in Albuquerque! I’ll be presenting our paper “The LLM Language Network” as an Oral tomorrow at 2:00 PM in Ballroom C, hope to see you there! Grateful to my co-authors Greta Tuckute, Antoine Bosselut, Martin Schrimpf — looking forward to the discussions!🧠

📢New Paper LLMs ace medical exams, but can they help real people? 👉 In our large study (1,298 participants), LLMs failed to improve how users identify medical conditions or choose care. 👉 Benchmarks ≠ real-world performance arxiv.org/abs/2504.18919 #AI #LLMs #MedTech

Excited to present tomorrow at the C3NLP workshop at #NAACL2025 our position paper: "Hire Your Anthropologist!" 🎓 Led by the amazing Mai Alkhamissi & Lorenzo Xiao, under the supervision of Mona Diab. Don’t miss it! 😄 arXiv link coming soon!