🔥The 9th Round of Easy Loan, Earn $40 Reward is in progress❗️ ⏰ Promotion Period: January 15th - Feburary 15th, 2025 👉 Register now and check more details at gate.io/campaigns/358

Announcing: Test queries for the TREC 2023 Deep Learning Track are now available, along with top-100s for (optional) use in reranking. #TRECDL #TREC2023DL MSMarco microsoft.github.io/msmarco/TREC-D…

🔔 Reminder: The deadline for run submissions for the TREC 2023 Deep Learning Track is in 1 WEEK! #TRECDL #TREC2023DL

... And... [drum roll] Paul Thomas Paul Thomas is coming to Tokyo in December to tell us all about it in his #ntcir17 keynote! @ntcir

Insights and industry stories for people interested in learning about metrics+ML in product!

🎉Happy to announce that we will be organising the Large Language Models for Evaluation in IR (#LLM4Eval) workshop at SIGIR 2025 2024. Stay tuned for more information! w/ Clemencia (Clem), Mohammad Aliannejadi, Nick Craswell, Charlie Clarke, Guglielmo Faggioli, Bhaskar Mitra | ভাস্কর মিত্র, Paul Thomas, and Emine Yilmaz

🎉 I am so excited to share that our paper "Synthetic Test Collections for Retrieval Evaluation" with Nick Craswell, Emine Yilmaz, Bhaskar Mitra | ভাস্কর মিত্র, and Daniel Campos has been accepted at #SIGIR2024! Stay tuned! Preprint and details will be available soon. #IR #LLM SIGIR 2025

RAG is all the RAGe these days, but we (still) don't quite know how to evaluate it properly... This year, we are taking a stab at it in the context of TREC, building on 30+ years of experience in evaluating IR systems. trec-rag.github.io

🥳🤩 LLMJudge ;) We have just released the LLMJudge challenge which is organised as part of the #LLM4Eval workshop at #SIGIR2024 (SIGIR 2025). Emine Yilmaz Nick Craswell Bhaskar Mitra | ভাস্কর মিত্র Paul Thomas Charlie Clarke Mohammad Aliannejadi Clemencia (Clem) Guglielmo Faggioli #IR #LLM #LLM4Eval #Evaluation

.Haolun Wu presenting our paper "Towards Group-aware Search Success" at ICTIR 2024. Preprint: arxiv.org/abs/2404.17313 #ICTIR2024

📢 Thrilled to announce that, "SynDL: A Large-Scale Synthetic Test Collection for Passage Retrieval" is publicly available. w/ Xi Wang, Emine Yilmaz, Nick Craswell, Bhaskar Mitra | ভাস্কর মিত্র, Paul Thomas Check it out: arxiv.org/abs/2408.16312

🎉Excited to share that our paper "SynDL: A Large-Scale Synthetic Test Collection for Passage Retrieval" has been featured in the Microsoft Research Focus Blog! 📜Check it out: microsoft.com/en-us/research… Xi Wang, Emine Yilmaz Bhaskar Mitra | ভাস্কর মিত্র, Nick Craswell Paul Thomas #IR #NLP #LLM

Wow, so happy to see our TREC RAG work featured in the *state of ai*! (here is the wonderful diagram summarized in ragnarok in slide 34 below)! 😲 Kudos to the team: Ronak Pradeep, Jimmy Lin, Nick Craswell, Daniel Campos, Shivani Upadhyay & NIST! 🍻 Thanks Nathan Benaich!

🔥Initial results from the TREC 2024 RAG Track! With respect to retrieval: our results suggest that automatically generated UMBRELA judgments can replace fully manual judgments to accurately capture run-level effectiveness. arxiv.org/abs/2411.08275

1/5🧵We the TREC RAG organizers, are excited to serve you the first course for the TREC 2024 RAG Track 🍽️ Brought to you by Shivani Upadhyay along with Nandan Thakur Daniel Campos Nick Craswell Ian Soboroff | [email protected] Hoa and Jimmy Lin, our 1st large-scale study on relevance assessments with an LLM!

💥 Next set of results from the TREC 2024 RAG Track! We've automated the nugget evaluation methodology using LLMs and obtained a strong correlation between automatic metrics and metrics derived from human annotations. arxiv.org/abs/2411.09607