🔥The 9th Round of Easy Loan, Earn $40 Reward is in progress❗️ ⏰ Promotion Period: January 15th - Feburary 15th, 2025 👉 Register now and check more details at gate.io/campaigns/358

If you want to understand: * How to choose the right setup for distributed training * Understand the costly mistakes that you should avoid * Ensure that you’re up to speed on the latest techniques for training at scale Myself and 14 other speakers will make it happen

Sometimes when you market, the original is just better

would add this little nugget overtrained models are hard to SFT overSFTed models are hard to RL overRLed models are hard to course correct with ICL feel free to verify

Yea and somewhat related sidenote: comparing h1/200s or b200s with 6000 adas or b6000s is crazy because not only are you losing vram, being capped on flops, you are also stuck with PCIe, and on top of that even, no NVLink. Its really a harsh division between their true datacenter

A few weeks ago Hugo Bowne-Anderson was kind enough to bring me on and yap about distributed training. It was a wonderful time! The episode is out now on all major platforms: youtube.com/live/76NAtzWZ2…

I’ll be presenting TorchTitan: a PyTorch native platform for training foundation models tomorrow at the ICML ES-FoMo@ICML2025 workshop! Come and say Hi!

When I left my local club of saltwater aquarium people (Zach lore; this is a real thing) to go start my world in tech, this is what their president said to me as I left. It shook me for a bit, but makes sense looking back

Merve giving us her secret sauce 👀

There won’t be any lighting lessons this week. Cooking up something special, stay tuned

The 🐐 s:

Not falling for OpenAI’s hype-vague posting about the new IMO gold model with “general purpose RL” and whatever else “breakthrough.” Google also got IMO gold (harder than mastering AIME), but remember, simple ideas scale best.

launching the training run by manually sshing into each node and starting the script because you cba to figure out how the cluster management software works

What currently are people’s solutions when you’ve run out of ChatGPT Deep Research’s for the month? It’s easier for me to get ChatGPT to actually adhere to my directions than Gemini in terms of output format. Just make like 3 accounts so you don’t pay $200/mo?

High-level tools like🤗transformers’ Trainer help you start fast, but long-term they can hurt. In this (free) series, I’ll share: 1. Why these abstractions stall growth 2. How low-level tools like accelerate fix that 3. And how they supercharge your high-level use later