🔥The 9th Round of Easy Loan, Earn $40 Reward is in progress❗️ ⏰ Promotion Period: January 15th - Feburary 15th, 2025 👉 Register now and check more details at gate.io/campaigns/358

Cool recent MLSys paper on MoE brought up by Deepak Narayanan at the ES-FoMo@ICML2025 workshop: arxiv.org/abs/2211.15841 ➡️ Casts MoE as block-sparse operations, enabling highly efficient GPU implementation.

Goddamn, wild times. Here's the paper you need to read understand today: arxiv.org/pdf/2211.15841… have fun.

A 8B-3.5T hybrid SSM model gets better accuracy than an 8B-3.5T transformer trained on the same dataset: * 7% attention, the rest is Mamba2 * MMLU jumps from 50 to 53.6% * Training efficiency is the same * Inference cost is much less arxiv.org/pdf/2406.07887

Thrilled to announce a new model trained jointly between Mistral AI and NVIDIA!

Excited to share some models we've been training over the last couple of months!