🔥The 9th Round of Easy Loan, Earn $40 Reward is in progress❗️ ⏰ Promotion Period: January 15th - Feburary 15th, 2025 👉 Register now and check more details at gate.io/campaigns/358

I am happy to share that our paper "𝐓𝐫𝐮𝐭𝐡 𝐊𝐧𝐨𝐰𝐬 𝐍𝐨 𝐋𝐚𝐧𝐠𝐮𝐚𝐠𝐞: 𝐄𝐯𝐚𝐥𝐮𝐚𝐭𝐢𝐧𝐠 𝐓𝐫𝐮𝐭𝐡𝐟𝐮𝐥𝐧𝐞𝐬𝐬 𝐁𝐞𝐲𝐨𝐧𝐝 𝐄𝐧𝐠𝐥𝐢𝐬𝐡" has been accepted in the #ACL2025 main conference. arxiv.org/abs/2502.09387…

Can you train a performant language models without using unlicensed text? We are thrilled to announce the Common Pile v0.1, an 8TB dataset of openly licensed and public domain text. We train 7B models for 1T and 2T tokens and match the performance similar models like LLaMA 1&2

BREAKING: Apple just proved AI "reasoning" models like Claude, DeepSeek-R1, and o3-mini don't actually reason at all. They just memorize patterns really well. Here's what Apple discovered: (hint: we're not as close to AGI as the hype suggests)

"Bad" data might be the secret sauce for "good" AI models. To make LLMs behave, let them first see how not to behave. This paper challenges the common wisdom in LLM pre-training that cleaner data always means better models, particularly concerning toxicity. Including toxic

simple "LLM as a judge" protip if you prompt for something like "provide answers to the TRUE/FALSE rubric questions in order, followed by a one sentence justification" this will be worse than the justification coming *before* the TRUE/FALSE marker

LLMs often struggle to balance memorizing training data with generalizing to new rules. This paper trains Transformer models of different sizes on simple tasks to show smaller models generalize but fail to memorize, while larger models memorize but fail to generalize, especially

Terence Tao reporting on the same problem we have seen endlessly in other domains: LLM’s produce output that “looks” correct but is often deeply wrong and even stupid on careful inspection. I know of no domain in which this is NOT the case. And yet people seem surprised over

Several folks reached out to me about this Apple paper on reasoning models. I have been saying this for the last 2.5 years: language models don't think, and they don't solve problems. They print sequences of words that make a *human reader believe* that there's a thinking or

Diffusion models create novel images, but they can also memorize samples from the training set. How do they blend stored features to synthesize novel patterns?  Our new work shows that diffusion models behave like Dense Associative Memory: in the low training data regime (number

GRPO quirk that contradicted my intuition: If you train on a group with rewards [0, 0, 0, 1] And then you train on another group with rewards [0.99, 0.99, 0.99, 1] Because of how GRPO normalizes within groups, the last trajectory will be equally reinforced in both cases!

Let's kick off our tutorial presentation by introducing our main sponsor: Hatedemics! This European project leverages AI to combat hate speech and misinformation. We're very grateful for the support! We can't wait to see you all in Vienna! #ACL2025NLP hatedemics.eu

Finally! A RAG over code solution that actually works (open-source). Naive chunking used in RAG isn't suited for code. This is because codebases have long-range dependencies, cross-file references, etc., that independent text chunks just can't capture. Graph-Code is a

For reasons unknown to me, the AI Safety community has put almost all of its eggs into scaling + system prompts + RL. Judging by how many problems we are seeing now (see below) with models built per that formula, shouldn’t we be desperately trying to find alternatives?

This paper is impressive! It introduces a clever way of keeping memory use constant regardless of task length. Great use of RL for AI agents to efficiently use memory and reasoning. Here are my full notes:

AI using AI as a source of information has made Google search full of errors and therefore useless. Google eliminated the most useful tool (especially for language workers) which was the number of hits for a phrase or a group of words. Instead, the AI search engine just makes up

i just reviewed five papers for NeurIPS and it was an awful experience: - first paper was clearly LLM-generated. it was too short, the references didn't work, had no experiments or theory at all, and a ton of obvious mistakes. the more i read the less it made sense - two were

So about a month ago, Percy posted a version of this plot of our Marin 32B pretraining run. We got a lot of feedback, both public and private, that the spikes were bad. (This is a thread about how we fixed the spikes. Bear with me. )

This: “Google eliminated the most useful tool (especially for language workers) which was the number of hits for a phrase or a group of words.” But also the rest of it.

I have been in Israel often in the past, especially many years ago for part of my PhD (Haifa). I had exactly this feeling, even as a stranger. You have to be there to understand. A melting pot of all walks of life and religions.

BREAKING: Explosive new paper from MIT/Harvard/UChicago. Things just got worse — a lot worse — for LLM’s and the myth that they can understand and reason. The paper documents a pattern they called Potemkins, a kind of reasoning inconsistency (see figure below). They show that