🔥The 9th Round of Easy Loan, Earn $40 Reward is in progress❗️ ⏰ Promotion Period: January 15th - Feburary 15th, 2025 👉 Register now and check more details at gate.io/campaigns/358

Gemstones: A Model Suite for Multi-Faceted Scaling Laws paper: arxiv.org/abs/2502.06857 code: github.com/mcleish7/gemst… The project analyzes scaling laws across 22 AI models, ranging from 50M to 2B parameters, by varying model width and depth.

Our new models for studying scaling laws are out! The Gemstones are 4K checkpoints (22 models) trained on 10T token combined, with varying architectures and learning rates. Here’s my fav new scaling experiment. It explains why industry has abandoned big dense models 🧵 (1/4)

Gemstones: A Model Suite for Multi-Faceted Scaling Laws McLeish et al.: arxiv.org/abs/2502.06857 #ArtificialIntelligence #DeepLearning #MachineLearning

Here are the top AI Papers of the Week (Feb 10-16): - Latent Reasoning - Large Memory Models - Brain-to-Text Decoding - Enhancing Reasoning to Adapt LLMs - Reinforcement Learning via Self-Play - Competitive Programming with Large Reasoning Models Read on for more:

If there is another McLeish/Schwarzschild duo doing recurrent deep learning out there, Avi Schwarzschild and I would love to meet you. Otherwise how are we dealing with the overload of ChatGPT made up references these days?! 🤯

people are talking about whether scaling laws are broken or pretraining is saturating. so what does that even mean? consider the loss curves from our recent gemstones paper. as we add larger models, the convex hull doesn’t flatten out on this log-log plot. that's good!

Multiple interesting experiments and findings for pretraining recipes. I especially liked the part about width/depth (not trivial to choose when you’re in the 1-3b range).

Today, we're joined by Jonas Geiping, research group leader at ELLIS Institute Tübingen and the Intelligent Systems to discuss his recent paper, “Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach.” This paper proposes a novel language model architecture which uses

#LLNL is not just advancing #AI, we are redefining how AI and science converge to unlock the next era of discovery. Huginn represents a new breed of language models that emphasizes careful introspection over immediate yet often incomplete answers. livermorelab.info/4ianNIM

Great article by Anil Ananthaswamy on latent reasoning systems coming out of AI at Meta, UMD Department of Computer Science , and ELLIS Institute Tübingen. quantamagazine.org/to-make-langua…

Excited to share our paper "Universal Sharpness Dynamics..." is accepted to #ICLR2025! Neural net training exhibits rich curvature (sharpness) dynamics (sharpness reduction, progressive sharpening, Edge of Stability)- but why?🤔 We show that a minimal model captures it all! 1/n

Before you leave Singapore be sure to check out the Tomlab's trio of pretraining papers at the Open Science for Foundation Models (SCI-FM) workshop in Hall 4 #5 ! Jonas and I will be around the rest of the afternoon to share our amd war stories 🥲

Update: 14 empirical papers added. 1. Learning to Execute. Wojciech Zaremba, Ilya Sutskever 2. Neural Programmer-Interpreters.Scott Reed, Nando de Freitas 3. Neural Programmer: Inducing Latent Programs with Gradient Descent. Arvind Neelakantan, Quoc V. Le, Ilya Sutskever 4.

Stronger models need stronger attackers! 🤖⚔️ In our new paper we explore how attacker-target capability dynamics affect red-teaming success (ASR). Key insights: 🔸Stronger models = better attackers 🔸ASR depends on capability gap 🔸Psychology >> STEM for ASR More in 🧵👇

Introducing ARGUS 👁️ A benchmark for measuring hallucinations and omissions in free-form captions generated by Video-LLMs.

Ever tried to tell if someone really forgot your birthday? ... evaluating forgetting is tricky. Now imagine doing that… but for an LLM… with privacy on the line. We studied how to evaluate machine unlearning, and we found some problems. 🧵

Big news! 🎉 I’m joining UNC-Chapel Hill as an Assistant Professor in Computer Science starting next year! Before that, I’ll be spending time OpenAI working on LLM privacy. UNC Computer Science UNC NLP