Repo to reproduce Grokking in a few lines of code (Full batch GD, small MLP, modular addition): github.com/okitouni/simpl…

🚀 Introducing the SEAL Leaderboards! We rank LLMs using private datasets that can’t be gamed. Vetted experts handle the ratings, and we share our methods in detail openly! Check out our leaderboards at scale.com/leaderboard! Which evals should we build next?

Empire State dragon??

You can just use a different model to prioritize higher signal tokens and generalize quicker. RHO-LOSS literally just works. (fraction is ratio of top-k tokens kept to total tokens; 1 is equiv to no rho-loss used)

Thesis from Ilya Sutskever : "to predict the next word, you have to predict the world" Antithesis from Yann LeCun : "AR-LLMs suck! Reversal curse!" Synthesis from FAIR: factorization-order-independent autoregressive model (MLM-U objective) (a paper subtweeting Mistral-PrefixLM thing?)

We eating good tonight

Friday feature drop: Highlight text or code within an Artifact and quickly have Claude improve or explain the selection.