🔥The 9th Round of Easy Loan, Earn $40 Reward is in progress❗️ ⏰ Promotion Period: January 15th - Feburary 15th, 2025 👉 Register now and check more details at gate.io/campaigns/358

So I run evaluation on Gemma 3 12B QAT vs. HQQ. HQQ takes a few seconds to quantize the model and outperforms the QAT version (AWQ format) while using a higher group-size. With GemLite bfp16 support, you can run quantized Gemma 3 faster without performance issues 🫡

Initial attempts on brining Gemlite to AMD MI300X

GemLite is significantly outperforming the default A16W4 vLLM kernel on the MI300X 🚀

GemLite: I feel the need, the need for speed!

Well optimized Triton kernels can perform very well end-2-end, even competing with highly optimized kernels like Marlin.

GemLite now supports vLLM vllm V1, which brings up to 1.25x faster inference speed vs V0! github.com/mobiusml/gemli…

We just made inference 1.5x faster with larger batch-sizes compared to last week 🤯 - work in progress

Wait, Triton can be faster than Cutlass ? 🧐

main Thien Tran Llama3.1 8B running A16W4 with FP16 accumulation on the 5090 RTX

GemLite 0.4.7 is out 🔥. It boosts performance by 5-10 tokens/sec end-2-end by using an interesting trick which might seem weird at first: sample the outputs from pre-allocated zeros. Explanation below 🧵 github.com/mobiusml/gemli…

Damn. Luckily, we have HQQ that only takes 5 secs to quantize an 8B model, super useful to get started right away with any model.

rocm support added 👀, mainly focusing on the MI300X github.com/mobiusml/gemli…

GemLite runs fast on the MI300X, but there's still plenty of performance left to unlock

FP4 weights meets high accuracy: Logit‐distillation bias correction for MXFP4 & NVFP4. On Llama-3.1-8B recovers ≥99% relative quality. Details at: mobiusml.github.io/fp4_blogpost/

Here's how to write a fast MXFP4/NVFP4 dequant GEMV kernel for batch-size=1: -Use a mapping + tl.gather to map the quant matrix block indices into double the fp4 range -Load the scales as e8m0.view(uint8) and use tl.exp2 to convert ~13% faster than SPLIT-K with tl dot_scaled

Simple and fast MXFP8 activation quant kernel: ✓ Padding-aware for arbitrary seq lens ✓ SM-aware unrolling to improve occupancy (large batches) ✓ ~10% faster than torch.compile in end-to-end inference