🔥The 9th Round of Easy Loan, Earn $40 Reward is in progress❗️ ⏰ Promotion Period: January 15th - Feburary 15th, 2025 👉 Register now and check more details at gate.io/campaigns/358

This is super exciting! While the early eval companies dove into meaningless metrics ("helpfulness"!), Quotient is targeting the evaluations that really matter. If your agent is lying to your customers, you need to know about it. What's more, you need to figure out why!

Announcing our speakers for the Retrieval + Search track! ⚠️PSA: Tix nearly sold out, get em here: ti.to/software-3/ai-…… Featuring: Aman, Former Founder, Harvey Jerry Liu, CEO, LlamaIndex 🦙 Julia Neagu, CEO, Quotient AI changhiskhan, CEO, LanceDB

We’re heading back to AI Engineer! Deanna Emery (founding AI Engineer at Quotient AI) and Maitar Asher 🎗️ (Head of Eng tavily) are speaking evaluating AI search. If you're building AI search, don't miss it.

Can’t wait to be back at AI Engineer! I’m teaming up with Maitar Asher 🎗️ from tavily to talk about evaluating AI search. We’re sharing a practical eval framework, lessons from real-world deployments, and never-seen-before benchmark results. Hope to see you there!

HypoEval evaluators (github.com/ChicagoHAI/Hyp…) are now incorporated into judges from Quotient AI — check it out at github.com/quotient-ai/ju…!

HypoEval is now available in Quotient AI's OSS judges! It uses SOTA hypothesis generation with just 30 human annotations to create decomposed rubrics, enabling LLMs to score criteria clearly. Beats fine-tuned models (w/ 3x more labels). Thanks Mingxuan (Aldous) Li for contributing!

detections go brrr One week in, Quotient AI Detections has processed 20M+ tokens, analyzed tens of thousands of logs, and caught thousands of hallucinations across real AI production apps. Still a long way to go, but we're committed to giving builders SOTA AI monitoring.

Most engineers think you need ground truth data to detect AI hallucinations. You don't. Extrinsic hallucinations are the real problem: model misusing the context you gave it. Here's a primer on how to do table stakes hallucination detection without expensive datasets👇

it was a pleasure speaking at AI Engineer with Maitar Asher 🎗️ from tavily and Deanna Emery from Quotient AI 🫡

retrieval + search track = best vibes AI Engineer ft Maitar Asher 🎗️ Deanna Emery Jerry Liu tavily Quotient AI LlamaIndex 🦙

Today's edition (8-Jun) of my newsletter is ready. (Consider subscribing, I write it daily. Link in comments & bio and also you will get a 1300+ page Python book as soon as you subscribe). Prompting with AI scales, Verifying doesn't

Do you need evals for your AI project? Freddie Vargus joins us this week to share his experience from Quotient AI and GitHub Co-pilot

“You want your model hitting milestones, not minefields.” Most AI eval talk is hand-wavy. This isn’t. Freddie Vargus (Quotient AI CTO) gets into the weeds: how to actually test tool use, avoid minefields, and build agents that don’t break. Check out the recording👇

Just shared the slides from our AI Engineer World Fair talk: Evaluating AI Search – A Practical Framework for Augmented Systems. As more AI agents rely on real-time data (like the web!), traditional eval approaches are falling behind and don't capture what's actually

AI Engineer Looking for more resources (think: research, OSS libraries, cookbooks and more!) for AI reliability? We have that! Check out Quotient AI Alpha, our collection of tool, resources and research. more coming weekly 👀

The worst part of building an agent? You don’t know it’s broken until your users tell you. We just dropped a cookbook for a web research agent with real time monitoring — so you can catch critical issues as they happen. ft. tavily LangChain OpenAI Quotient AI

What did Freddie Vargus see? 👀 Everyone’s talking about context engineering now. Freddie knew months ago: context is the product.

how do i catch hallucinations? come learn to implement monitoring systems that catch AI errors as they happen in live production environments with Julia Neagu and Quotient AI if you register, you'll be sent the recording and study notes after they're done!

If you're shipping LLMs to production and still finding out about critical from your users, this course is for you. Real-time evals, automated detection, and the tools we use at Quotient AI to keep AI grounded. On July 30th jason liu and myself are laying it all out.

DMs OPEN for topics you want covered. I write my talks the night before. it's a really bad habit. it stresses out Deanna Emery