⚗️ UMBRELA qrels for the TREC 2024 RAG Track have been posted! arxiv.org/abs/2411.08275

🎙️ We bring to you UMBRELA ⛱️ relevance judgments for the TREC 2024 RAG topics! It should be a good proxy for evaluating your retrieval pipelines with the TREC 2024 RAG test topics. New topics to assess your systems! In time, other judgments shall be made public too. 🧙‍♂️

As Ian Soboroff | [email protected] notes kicking off TREC 2024, the 33rd iteration of the series, this is the 30th anniversary of BM25, introduced at TREC-3: trec.nist.gov/pubs/trec3/t3_…

2024 has been a fun year! Initial collab with Vectara on multilingual RAG benchmarking to coordinating a first-ever RAG track (TREC RAG @ 2025) in TREC 2024 and lastly getting to intern at Databricks Mosaic Research! I predict 2025 will be the year of diversity & realistic evals in IR & RAG!

Now we know RAG is alive (again). I'll be talking about evals this, evals that, evals!

Hello world! I'm back for TREC RAG 2025. Did you miss us?

We had a successful participation of over 45+ teams & 150+ runs last year in TREC RAG 2024! 🔥🔥 We are back with TREC RAG @ 2025 2025 this year! Make sure you participate in one of the RAG tracks this year to help accelerate RAG evaluation! ⚡⚡ Signup here: docs.google.com/forms/d/e/1FAI…

📢📢RAG 2025 topics are officially now released! 🔍Test narratives are out now (total 105): trec-rag.github.io/annoucements/2… Let the games begin! #TREC2025 #RAG

We have released the TREC RAG @ 2025 2025 topics and will be out with strong baselines soon. But for those who are eager, go right ahead!

TREC RAG TOPICS ARE OFFICIALLY OUT NOW!!! 🔥🔥🔥 LET THE GAMES BEGIN! 🥶

We are out with the official baselines for TREC RAG @ 2025 this year: github.com/castorini/ragn… Shivani Upadhyay and I had fun putting together a strong Retrieve (Pyserini) -> Rerank (RankLLM) -> Augmented Gen (Ragnarök) baseline and we hope to see you all beat it!

🚨 TREC RAG 2025 update: Official baselines are now available! 📅 Mark your calendars: submissions due August 17

TREC RAG 2025 official retrieval baselines are available now! 💥💥💥 Time to start generating those answers and submit them to eval base before August 17th! 🗓️ Let the games begin, well you have less than a month remaining to submit! 🍻