🔥The 9th Round of Easy Loan, Earn $40 Reward is in progress❗️ ⏰ Promotion Period: January 15th - Feburary 15th, 2025 👉 Register now and check more details at gate.io/campaigns/358

Existing RAG benchmarks rely on automatic models for support, but can LLMs potentially replace human assessors? 📚 Our new large-scale study compares human vs LLM judges with 11K human annotations, 36 topics, 45+ RAG answers in the TREC 2024 RAG Track. To appear in #SIGIR2025⚡️

🚨Attn #NLProc and friends! 🚨 Our shared task on learning to contextualize scientific claims is back! We have some exciting new updates that make the task harder and the evaluation more thorough! Made w Joel Chan | 🦣: [email protected] and Matthew Akamatsu (1/4)

Evaluating RAG fact recall is crucial but manual eval can be slow! Can LLMs reliably automate fact extraction & eval for RAG systems? 📚 Our new large scale study, The Great Nugget Recall, provides extensive analysis using the AutoNuggetizer framework on TREC RAG @ 2025! In #SIGIR25🇮🇹

AI agents are entering online social spaces, but often their messages feel generic or intrusive. In our #CHI25 paper, we introduce Social-RAG, a workflow that grounds AI generations in the specific group context by retrieving from the group’s interaction history. 🧵(1/9)

We’re live on Reddit! Ask us Anything about our OLMo family of models. We have six of our researchers on hand to answer all your questions.

[Blogged] Ai2 Paper Finder and Futurehouse PaperQA2: More transparent Deep Search for Scholars? musingsaboutlibrarianship.blogspot.com/2025/05/ai2-pa…

Ai2 Semantic Scholar is hiring an #ml #nlp #ai reasoning researcher for a Research Scientist, Agents for Science position with target start dates in 2025. Excited about developing AI systems with deep reasoning capabilities for science? Send an application our way!

You can set `UV_TORCH_BACKEND=auto` and uv will automatically install the right CUDA-enabled PyTorch for your machine, zero configuration

We're hiring an #ML #NLP #AI reasoning researcher for a Research Scientist, Agents for Science position! Excited about developing AI systems with deep reasoning capabilities for science? 🧑‍🔬

OpenAI I think "AI" has mostly affected me in two ways: (1) I spend less time trying to remember / explore / search my way into the correct parameters for bad APIs (eg matplotlib, seaborn, some shell commands) (2) I have a low-sensitivity / high specificity literature search assistant

Why can AIs code for 1h but not 10h? A simple explanation: if there's a 10% chance of error per 10min step (say), the success rate is: 1h: 53% 4h: 8% 10h: 0.002% Toby Ord has tested this 'constant error rate' theory and shown it's a good fit for the data chance of

Many people have been asking for an interface to OpenHands that is: 1. easy to install (no docker) 2. can be used in your standard development environment This new CLI checks both of these boxes, and is fun to use!

New updates for olmOCR, our fully open toolkit for transforming documents (PDFs & images) into clean markdown. We released: 1️⃣ New benchmark for fair comparison of OCR engines and APIs 2️⃣ Improved inference that is faster and cheaper to run 3️⃣ Docker image for easy deployment

I have to admit that it’s cool to see an AI lab that I look up to use retrieval technology built in Trondheim

you’re on cursor? you’re still using windsurf? you might as well be on github copilot. everyone’s on aider. we’re all using zed. we’re now on open hands. open hands is for losers, just kidding we’re using cline. we’re on roocode. we’re hand rolling our own claude code cli clone.

"if you define jobs in terms of tasks maybe you're actually defining away the most nuanced and hardest-to-automate aspects of jobs, which are at the boundaries between tasks."

who would win? 1) Claude writing code for hours, downloading gigs of data, resulting in the most cursed analysis known to man 2) A human looking at the data for 5 minutes I am sad to admit that 1) led to 2) and will now start form scratch

Introducing SciArena, a platform for benchmarking models across scientific literature tasks. Inspired by Chatbot Arena, SciArena applies a crowdsourced LLM evaluation approach to the scientific domain. 🧵

i'm really going out on a limb here saying I like health & wellness, normal circadian rhythms, and would resign if I built a Nazi bot

Firstly, I think AI speedup is very weakly correlated to anyone's ability as a dev. All the devs in this study are very good. I think it has more to do with falling into failure modes, both in the LLM's ability and the human's workflow. I work with a ton of amazing pretraining