starting in <45 min - faster-llms.vercel.app

🚀 Calling all OSS contributors to vLLM, SGLang, HF Transformers, TGI, TRT-LLM & more — Join us at the Inference Engines Social @ ICLR 🇸🇬 📍 Singapore EXPO 🗓️ Apr 25, 13:00 Let’s hang out, swap ideas & share what we’re building! 🔗 lu.ma/42pobekx

🚀 Vercel’s #StateOfAI survey highlights: 83% of teams run ≤3 models, 86% don’t train their own, and 46% say the biggest hurdle is latency or cost (23% latency + 23% cost). Devs are leaning hard on third‑party inference—OpenAI alone powers 83% of workloads

Hiding the verification latency of speculative decoding = faster inference. Come chat with me about how! 📍 Poster #245 (Hall 3) 🕒 Today, 15:00–17:30

Twitter! I can't believe no one told me that people are using vision encoders to retrieve document information these days. Based on a tip from Nadav Timor I read the "ColPali: Efficient Document Retrieval with Vision Language Models" paper, and it is very cool.

🎉 #icml2025 spotlight: our speculative decoding verification methods for heterogeneous vocabs!

🎉 #icml2025 oral (top 1%) – our speculative decoding paper: arxiv.org/abs/2502.05202

Do you want to speed up your inference time by 2X for free? Using speculative decoding (SP) but want to use your own drafter? Want lossless speedup with one line? Go to Nadav Timor talk at ICML tomorrow at 3:30 PM and check out his poster afterward at 4:30 PM. Thank me later!

Algorithms developed by researchers in Prof. David Harel’s team and Intel Labs enable AI developers around the world to combine the power of different AI models “thinking” as one. The study led by PhD student Nadav Timor was presented at #ICML2025>> bit.ly/faster-ai-eng

KV cache go brrr with Junchen Jiang's LMCache Lab! Join us tomorrow to learn more about next‑gen long‑context LLM inference: faster-llms.vercel.app

I’ll be joining the GPU MODE hackathon 🎉 See you there!

In inference, speculation parallelism sustains high MFU by hiding DP comms bottlenecks

Local inference in your vscode is here