Sifflet raises cash to expand its data observability platform tcrn.ch/3Z1IzAD

Since I got asked multiple times this week. Here is a fully self contained LLaMA2 built for ROCm/AMD from a Macbook with ZML. You can copy and unpack this tar on a machine _without_ ROCm installed and it will run on the GPU.

Hey folks, since we've been asked so many times, here is a quick demo of what we're building. This is a small LLaMA2 sharded on 1 NVIDIA RTX 4090 (in Paris), 1 AMD 6800XT (in Corendos's flat) and 1 Google Cloud TPU v2 over Tailscale. Exact same code, all built on my Mac

Ticketmaster playing my favourite Oasis album

github.com/zml/zml

It’s officially out!

ZML: a high-performance AI inference stack that can parallelize and run deep learning systems on lots of different hardware. It's out of stealth, impressive, and open source.

We can finally say it publicly: poolside got their models running on Amazon Web Services’s Trainium/Inferentia thanks to ZML. No code change. Promise kept.

zml.ai

We've been cooking with the ZML team. We just dropped a new website to showcase what we do best: fast, robust and scalable inference. Check it out 👉 zml.ai

Mitchell Hashimoto Tristan Ross 😺❄️ Bun Literally yesterday with Francis Bouvier and Katie Hallett from Lightpanda

boy oh boy am I excited for this. Steeve Morin from ZML demoing at AI Demo Days lffffgggg one server. many models. speed afffff

👀👀🧑‍🍳

Great to see these two together Steeve Morin & Harry Stebbings🔥🔥

hub.docker.com/r/zmlai/llmd

The tech preview of LLMD is out: - Easy Setup - Just mount your model and run - Cross-Platform GPU Support - Single container works on *both* NVIDIA and AMD GPUs - Lightweight - Only 2.4GB container size - High Performance Enjoy !

Hardware independent LLM inference engine from ZML.

The ZML team and Steeve Morin cooked a new high-performance LLM inference engine. - lightweight 2.4GB container - easy on cross-platform - written in zig I just tested it and deployed on Hugging Face Inference Endpoints. You can try it out in 5min! 👇

The fastest LLM server is now faster with ROCm 6.4.1 and Tri Dao's newer flashattn. There are some cool packaging features on that image. First, 50mb compressed smaller. Second, through careful layering, it overlaps download and faster extraction. Complete Docker pull is ~20s.

And after 1 week of work, here is zml/llmd running transparently on TPU with full prefill/decode paged attention. No code change, single flag, as it should be.