Listing issues in NLG evaluations turned into a 25 page survey! In “Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text”, Thibault Sellam Elizabeth Clark and I cover 250+ papers. 📄Link: arxiv.org/abs/2202.06935 Want to learn more?👇

My colleagues Chris Alberti, Kuzman Ganchev and I are looking for a fall intern. The topic is improving language technologies for underrepresented languages by leveraging large pretrained language models. The position could be in NYC or remote. If interested, please reach out.

Currently presenting MultiBERTs at ICLR, we will be at Poster Session 5 until 12.30pm Pacific

New from Google Research! Language models perform amazing feats, but often still "hallucinate" unsupported content. Our model, RARR🐯, automatically researches & revises the output of any LM to fix hallucinations and provide citations for each sentence. arxiv.org/abs/2210.08726 🧵

Amos is a new optimizer that we propose to pre-train large language models. It is more efficient and converges faster than AdamW: ≤ 51% memory for slot variables, and better valid loss within ≤ 70% training time! New from Google Research. Preprint: arxiv.org/abs/2210.11693

We are excited to release Seahorse 🌊🐴, a ✨multilingual, multifaceted summarization evaluation dataset✨ 96,000+ human ratings to enable faster progress in training and evaluating learnt metrics for summarization! Preprint: arxiv.org/abs/2305.13194 Data: goo.gle/seahorse

Learn how SQuId (Speech Quality Identification), a 600M parameter regression model that describes to what extent a piece of speech sounds natural, can be used to complement human ratings for the text-to-speech evaluation of many languages → goo.gle/3J1kghg

🚨 Dream Job Alert 🚨 We are looking to fill various positions including NLP researcher, data engineer and software developer. If you are interested in #LLM, causal inference and having a positive impact on the world, please reach out! worldbank.org/en/research/di…

The Searhorse dataset is available - 96K ratings to train and evaluate new summarization metrics. Congrats Elizabeth Clark and team! Paper here: arxiv.org/abs/2305.13194 Models here: huggingface.co/collections/go…

I am glad to share what we've been up to for the past few months ♊✨

Super excited for this launch! Had a great time working with a really amazing team getting this out!! Check it out:

Excited to share new work from Google DeepMind: “ProtEx: A Retrieval-Augmented Approach for Protein Function Prediction” biorxiv.org/content/10.110…

Google presents Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More? Long-context LM: - Often rivals SotA retrieval and RAG systems - But still struggles with areas like compositional reasoning repo: github.com/google-deepmin… abs: arxiv.org/abs/2406.13121

The playlist starts in West Africa and wanders West-to-East until it hits Brazil. It’s composed of songs supplied by native speakers, which tend to be bangers. youtube.com/playlist?list=…

Making LLMs run efficiently can feel scary, but scaling isn’t magic, it’s math! We wanted to demystify the “systems view” of LLMs and wrote a little textbook called “How To Scale Your Model” which we’re releasing today. 1/n