🔥The 9th Round of Easy Loan, Earn $40 Reward is in progress❗️ ⏰ Promotion Period: January 15th - Feburary 15th, 2025 👉 Register now and check more details at gate.io/campaigns/358

Could large language models (LLMs) replace humans for relevance judgment tasks? #LLMJudge aims to explore this question by examining various factors, including the size of the LLM, the prompt design, and other relevant parameters. #LLMJudge LLM4Eval Workshop #LLMs #judgment #SIGIR

Currently, ongoing interesting discussion on the LLM Challenge from participants. #Sigir

Great panel discussion with Ian Soboroff | [email protected], L. Dietz @[email protected] @[email protected], Charlie Clarke, Michael Ekstrand moderated by Bhaskar Mitra | ভাস্কর মিত্র at LLM4Eval Workshop. #SIGIR24

📢 Thrilled to announce that, "SynDL: A Large-Scale Synthetic Test Collection for Passage Retrieval" is publicly available. w/ Xi Wang, Emine Yilmaz, Nick Craswell, Bhaskar Mitra | ভাস্কর মিত্র, Paul Thomas Check it out: arxiv.org/abs/2408.16312

🔊 The second workshop on Large Language Models for Evaluation in IR (#LLM4Eval), co-located with #WSDM2025, is now accepting submissions! If your work explores or leverages LLM capabilities for evaluation, we encourage you to submit and share your research.

The call for papers is now open for the second workshop of #LLM4Eval: 💡 See CFP: llm4eval.github.io/WSDM2025/cfp/ 🗓 Submission deadline: January 20, 2025

Organiser: Hossein A. (Saeed) Rahmani, Clemencia (Clem), Mohammad Aliannejadi, nick r, Charlie Clarke, Guglielmo Faggioli, Bhaskar Mitra | ভাস্কর মিত্র, Paul Thomas , and Emine Yilmaz

The workshop will focus on models, techniques, data collections, and methodologies for information retrieval evaluation in the era of LLMs. #WSDM2025 #IR #Evaluation #Performance #Workshop #NLP #SIGIR

In today's 2nd Workshop on #LLMs for Evaluation in #InformationRetrieval (#LLM4Eval) at #WSDM2025 (09:15 CET), Edgar Meij, our Head of #AI Platforms Engineering, will deliver a keynote on "Synthetic Evaluations & #GenAI Application Development" bloom.bg/4hyocEi #AI

We will host multiple breakout discussions to pick your brains 🧠 on the future challenges and research on this topic. Let's shape the LLMEval research agenda. Interested? Already have thoughts and ideas? Then, submit your extended abstracts or just join us in the workshop.

The deadline for submitting your LLM evaluation paper is tomorrow. Looking forward to interesting contributions!