🔥The 9th Round of Easy Loan, Earn $40 Reward is in progress❗️ ⏰ Promotion Period: January 15th - Feburary 15th, 2025 👉 Register now and check more details at gate.io/campaigns/358

While most papers on knowledge distillation focus on student accuracy, we investigate the agreement between teacher and student networks. Turns out, it is very challenging to match the teacher (even on train data!), despite the student having enough capacity and lots of data.

Excited to kick-start focus #SciML series on #ML meets Info theory and statistical mechanics! Amazing speaker/session chair line-up: Alex Alemi (Max Welling), @pratikac (Karthik), Sho Yaida (Jascha Sohl-Dickstein), Yasaman Bahri (Surya Ganguli) and Elena Agliari. Details at: cmu.edu/aced/sciML.html

We are presenting our paper "Does Knowledge Distillation Really Work?" at #NeurIPS2021 poster session 2 today - come check it out! Joint work with Pavel Izmailov, Polina Kirichenko, Alex Alemi, and Andrew Gordon Wilson. Poster: nips.cc/virtual/2021/p… Paper: arxiv.org/abs/2106.05945

A pretty cool paper (and I also hope useful) on using pre-training models to create highly informative priors for downstream tasks. Thanks to all the collaborators, it was a lot of fun!

We are thrilled to announce Imagen, a text-to-image model with unprecedented photorealism and deep language understanding. Explore imagen.research.google and Imagen! A large rusted ship stuck in a frozen lake. Snowy mountains and beautiful sunset in the background. #imagen

1/ Super excited to introduce #Minerva 🦉(goo.gle/3yGpTN7). Minerva was trained on math and science found on the web and can solve many multi-step quantitative reasoning problems.

Want to understand and/or play with variational diffusion models? - See colab.research.google.com/github/google-… for a simple stand-alone implementation and explanation. (Thanks Alex Alemi and Ben Poole for making this)! - See colab.research.google.com/github/google-… for an even more basic implementation on 2D data.

Durk Kingma Ben Poole To accompany the colab, I've also written a blog post blog.alexalemi.com/diffusion.html attempting to make sense of the VDM Diffusion loss. In it, I try to motivate how the VDM diffusion loss is simply the joint KL between the forward and reverse process.

Happy to announce DreamFusion, our new method for Text-to-3D! dreamfusion3d.github.io We optimize a NeRF from scratch using a pretrained text-to-image diffusion model. No 3D data needed! Joint work w/ the incredible team of Ben Mildenhall Ajay Jain Jon Barron #dreamfusion

PaLM 540 Billion, Google's large language model used 4.2 moles of flops to train. 4.2 Moles!

Each delivery service should use its own distinctive knock.

Ever wonder why we don’t train LLMs over highly compressed text? Turns out it’s hard to make it work. Check out our paper for some progress that we’re hoping others can build on. arxiv.org/abs/2404.03626 With Brian Lester, Jaehoon Lee, Alex Alemi, Jeffrey Pennington, Adam Roberts, Jascha Sohl-Dickstein

Is Kevin onto something? We found that LLMs can struggle to understand compressed text, unless you do some specific tricks. Check out arxiv.org/abs/2404.03626 and help Jaehoon Lee, Alex Alemi, Jeffrey Pennington, Adam Roberts, Jascha Sohl-Dickstein, Noah Constant and I make Kevin’s dream a reality.

In which I try to make sense of most of machine learning: blog.alexalemi.com/kl-is-all-you-…

Why don't we measure probabilities in degrees? blog.alexalemi.com/a-degree-of-ce…

If you miss the NYTimes needle, especially one that is statistically uniform (blog.alexalemi.com/a-degree-of-ce…), you can use this page: alexalemi.com/random/electio… I whipped together to reason about the correlations between the swing states tonight as results come in.

I am recruiting Ph.D. students for my new lab at New York University! Please apply, if you want to work with me on reasoning, reinforcement learning, understanding generalization and AI for science. Details on my website: izmailovpavel.github.io. Please spread the word!

Recently I've been playing around with a quarter-order-of-magnitude system for simple calculations. It gives better precision than single sig-fig calculations using only four, very intuitive, symbols. blog.alexalemi.com/quarters.html