🔥The 9th Round of Easy Loan, Earn $40 Reward is in progress❗️ ⏰ Promotion Period: January 15th - Feburary 15th, 2025 👉 Register now and check more details at gate.io/campaigns/358

We have day 0 support for #Qwen3 by Alibaba Qwen on Baseten using SGLang. Qwen 3 235B's architecture benefits from both Tensor Parallelism and Expert Parallelism to run Attention and Sparse MoE efficiently across 4 or 8 H100 GPUs depending on quantization. More in 🧵

There's a lot to be excited about with Qwen 3: - Fits on 4xH100 - 1/4 the cost of DeepSeek-R1 in production - "Hybrid thinking" makes reasoning optional - Continues the Qwen tradition of being great at coding Deployment w/ SGLang + vibe check in the video!

Early benchmarks of Qwen 3 with SGLang show promising initial results and key avenues for improvement. We're seeing: - Up to 76 TPS per user for real-time - Up to 4600 total token throughput for batch - 32 concurrent requests as a good balance for prod Details in 🧵

People told us they want Orpheus TTS in production. So we partnered with Baseten as our preferred inference provider! Baseten runs Orpheus with: •⁠ ⁠Low latency (<200 ms TTFB) •⁠ ⁠High throughput (up to 48 real-time streams per H100) •⁠ ⁠Secure, worldwide infra

I’ll be joining my Baseten colleague Philip Kiely at the AI Engineer World’s Fair AI Engineer in San Francisco, June 3–5, to Introduce LLM serving with SGLang LMSYS Org. We’d love for you to stop by and exchange ideas in person!🤗

My two favorite log lines.

Great chatting all things voice agents with kwindla today in his course! Main takeaway: infra problems > GPU problems for voice. 1. Network overhead between client & each model 2. Client code (streaming/websockets/sessions) 3. But STT/TTS optimization w/ TRT-LLM matters too

It's now 10x easier to get started building with open source models.

let there be inference

Announcing our speakers for the Voice track! ⚠️PSA: Tix nearly sold out, get em here: ti.to/software-3/ai-…… Featuring: kwindla, CEO Daily Sean DuBois, WebRTC and Realtime API OpenAI Brooke Hopkins, Founder coval @dkundel, Developer Experience OpenAI

Going from off-the-shelf to top-shelf.

Everything you need to know about running inference in production in just one PDF.

Replacing OpenAI o3 with DeepSeek R1 0528 and saving >85% on token costs.