🔥The 9th Round of Easy Loan, Earn $40 Reward is in progress❗️ ⏰ Promotion Period: January 15th - Feburary 15th, 2025 👉 Register now and check more details at gate.io/campaigns/358

New paper out 🚀 Zero-shot Benchmarking: A Framework for Flexible and Scalable Automatic Evaluation of Language Models: arxiv.org/abs/2504.01001. We present a framework and release a repository for creating reliable benchmarks for (V)LM tasks quickly and fully automatically.

We just released M-Prometheus, a suite of strong open multilingual LLM judges at 3B, 7B, and 14B parameters! Check out the models and training data on Huggingface: huggingface.co/collections/Un… and our paper: arxiv.org/abs/2504.04953

Come and chat with us about a powerful (but surprisingly underused) *test-time compute* scaling technique to improve your LLMs!

👉 slator.ch/QuestionAnswer… Two new research papers propose using question answering ❓ to evaluate #AI #translation, 🤖 challenging 🧐 how the language industry evaluates translation quality. Dayeon (Zoey) Ki UMD CLIP Marine Carpuat Johns Hopkins University Patrick Fernandes Language Technologies Institute | @CarnegieMellon Instituto Superior Técnico

Very proud of this work with Patrick Fernandes Sweta Agrawal Manos Zaranis Graham Neubig at Sardine Lab and Language Technologies Institute | @CarnegieMellon. TL;DR: We evaluate translation quality of complex content by checking question answering invariance.