Can we increase the efficiency *and* performance of auto-regressive models? We introduce dynamic-pooling Transformers, which jointly perform language modelling and token segmentation. Piotr Nawrot* Adrian Lancucki Jan Chorowski 📜arxiv.org/abs/2211.09761 🧑‍💻github.com/PiotrNawrot/dy…

Introducing *nanoT5* Inspired by Jonas Geiping's Cramming and Andrej Karpathy's nanoGPT, we fill the gap of a repository for pre-training T5-style "LLMs" under a limited budget (1xA100 GPU, ~20 hours) in PyTorch 🧑‍💻github.com/PiotrNawrot/na… EdinburghNLP

The memory in Transformers grows linearly with the sequence length at inference time. In SSMs it is constant, but often at the expense of performance. We introduce Dynamic Memory Compression (DMC) where we retrofit LLMs to compress their KV cache while preserving performance

Tomorrow at ICML Conference, together with Edoardo Ponti and Adrian Lancucki, we'll present an updated version of "Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inference". You can find an updated paper at arxiv.org/abs/2403.09636. Among others - 1) We trained DMC to

🚀 By *learning* to compress the KV cache in Transformer LLMs, we can generate more tokens for the same compute budget. This unlocks *inference-time hyper-scaling* For the same runtime or memory load, we can boost LLM accuracy by pushing reasoning even further!

TokShop videos are finally out! 🎥🤩 Check out the great talks from Yuval Pinter (Join them? beat them? Fix them?) Desmond Elliott (Pixel LM) Adrian Lancucki (dynamic segmentation) . panel with hot takes from 🔥: Alisa Liu @ COLM 🦙 Albert Gu Yuval Pinter Sander Land Kris Cao