🔥The 9th Round of Easy Loan, Earn $40 Reward is in progress❗️ ⏰ Promotion Period: January 15th - Feburary 15th, 2025 👉 Register now and check more details at gate.io/campaigns/358

Rumor is 2>1 For clarifications ask Peano

Ok there's a new paper in my top 3 favorites Vision transformers need registers Clear problem, elegant solution, well written, easy to understand, good results, limitations included. No fancy losses or layers. No equation (at all!) Here's a short summary: (1/4)

I also view layernorm as hyperplane proj + hypersphere proj Hyperplane proj makes no sense, hence we do RMSnorm now Although don't forget the epsilon. We project onto the hyper*ball* actually

Summary of "Massive activations in LLMs": - "artifact" tokens are in all transformers, ViTs and LLMs - their weirdness is ~only on 1 channel - they are the same as the quantization outliers - their purpose is *not* global information - there's a fix simpler than registers

Oh these plots are great too They fit with my observations on norms of different things

Is there a good reason we use softmax losses in contrastive learning, instead of just doing MSE? ie L = ||xi-xi'||² - lambda sum_k ||xi-xk'||² I'd guess the optimization dynamics are maybe friendlier, but does anyone have a good pointer? Both for CLIP and SSL btw

Two things teach intellectual humility: people smarter than you, and maths. Doing math with people smarter than you is sort of a bit too much.

So the reason I was asking about this is because the squared L2 has the very pleasant property of reducing to "just push away from the avg" and that would eliminate all batch size issues (you an use an EMA avg) It's basically what DINO does, w/ softmax+CE loss instead of L2

Vision transformers have high-norm outliers that hurt performance and distort attention. While prior work removed them by retraining with “register” tokens, we find the mechanism behind outliers and make registers at ✨test-time✨—giving clean features and better performance! 🧵

Great summary of dino.txt by Fede! Drop by the poster if you're at CVPR! 📅 Sunday, June 15 🕥 10:30 - 12:30 📍 Poster 370

FFS Hugging Face please stop doing that it makes you look like pretentious assholes

In case there is any ambiguity: DINOv2 is 100% a product of dumb hill-climbing on ImageNet-1k knn accuracy (and linear too) Overfitting an eval can be bad. But sometimes the reward signal is reliable, and leads to truly good models. It's about finding a balance

I already know who Jiahui is you don’t have to tell me

Hey I'm a doctor now, neat

~400 people have joined us on Sunday at Cohere Labs Open Science Community ML Summer School. TimDarcet as always, delivering a super amazing talk on Scaling Self Supervised Learning (SSL, Dinov2, Masked Image Modeling, CAPI) Super interesting session.

Why does Meta open-source its models? I talked about it with Maciej Kawecki - This Is IT looking at Dino, our computer vision model with applications in forest mapping, medical research, agriculture and more. Open-source boosts AI access, transparency, and safety. youtube.com/watch?v=eNGafi…