🔥The 9th Round of Easy Loan, Earn $40 Reward is in progress❗️ ⏰ Promotion Period: January 15th - Feburary 15th, 2025 👉 Register now and check more details at gate.io/campaigns/358

We took a deep dive into the DeepSeek R1 tech report today at Hugging Face and recorded the discussion :) Let me know if you'd like us to publish our journal club more often! youtu.be/1xDVbu-WaFo

Thrilled to share that our work No Need to Talk: Asynchronous Mixture of Language Models [arxiv.org/abs/2410.03529] has been accepted to #ICLR2025! In this paper, we explore strategies to mitigate the communication cost of large language models, both at training and inference,

DeepSeek R1 (the full 680B model) runs nicely in higher quality 4-bit on 3 M2 Ultras with MLX. Asked it a coding question and it thought for ~2k tokens and generated 3500 tokens overall:

Deepseek V3 and R1 discourse boils down to this. Shifting the curve means you build more and scale more dummies

Someone please explain Jevons paradox to Wall Street

The DeepSeek V3 model file is ~450 lines of code in MLX LM. Includes pipeline-parallelism and all. Good way to see how it all works.

🚨 One question that has always intrigued me is the role of different ways to increase a model's capacity: parameters, parallelizable compute, or sequential compute? We explored this through the lens of MoEs:

“You don’t look grateful.”

I'm going to catch hell for posting but to summarize: 1. This paper misled its way to an #ICLR2025 Oral 2. I pointed this out 3. AC rejected the paper 4. Authors complained & somehow persuaded ICLR to overrule the AC and award a Spotlight 5. AC made clear they were overruled