🎄 Wishing everyone a very happy Christmas from LAMA-Lab! 🎅🔬 We've had an incredible time together last few days, feasting at our potluck and diving into a hackathon to craft our very own language model from the ground up! Merry Christmas and happy holidays to all! 🎁🌟

This is the ChemBench core team (Adrian Mirza, Nawaf Alampara, sreekanth kunchapu, Kevin M Jablonka) trying to crash another leaderboard 🎳.

🚀 Excited to share our debut project from Jablonka Lab (Lab for AI for Materials), ChemBench. Dive in for more insights: we've compared all the LLMs! #ChemBench #Teamwork #llm #ai4science #Chemistry

Can LLMs model materials? Checkout our latest work MatText, we explore some of the interesting and crucial questions in modelling materials with LLMs 🤓

1/8 📊 Materials science has a wealth of untapped data in papers, but lacks curated datasets for #ML and innovative materials design.

For benchmarks, too, less is more. We are deleting a lot of the next version of ChemBench. Deleting questions lets us focus more on difficult evals and makes the benchmark less noisy. But creating difficult questions takes a lot of discussions ...

Cats are invariant under SO(3) transformations! 😼

🚨ChemBench Update🚨 We have been hard at work over the past few weeks improving our benchmark for evaluating LLMs in chemistry. Find a new version on arXiv! arxiv.org/abs/2404.01475

I'm super proud of the team for the massive effort that went into this revision. It has been the most intense revision I ever worked on but we think the work improved a lot: a new human baseline (with tool use), many more hand-curated advanced questions, deeper analyses ...

Chemists often combine many different techniques to elucidate structures. Adrian Mirza has been building a system that mimics this using machine-learning models and genetic algorithms.

🚀Our revised MaCBench paper is now on arxiv! arxiv.org/pdf/2411.16955 Key updates! 🌟Robust reproducibility: 5x experiment runs + error bars for statistical confidence 🌟Full dataset & leaderboard: Now live on HuggingFace with model comparisons huggingface.co/spaces/jablonk…

Many of our benchmarks underwent a large revision in the last weeks. We now host HuggingFace spaces for them and the dataset. The revised article and post below give more details

🚀ChemBench just leveled up! We’re thrilled to announce the latest release of ChemBench—now smarter and smoother! Dive into benchmarking any chemistry AI model with our revamped framework, designed for flexibility and ease. #ChemistryAI #MachineLearning #OpenScience #Innovation

Day 1 of the Foundation Models workshop hosted by the ELLIS Winter School!

LAMA Lab at ICLR in Singapore! Come check out our posters at the AI4MAT workshop on Monday sites.google.com/view/ai4mat/ho… #ICLR2025 #Singapore #MachineLearning #Chemistry #ArtificialInteligence

We're excited to present our posters at the AI4Mat Workshop today at #ICLR25 #AI4Mat #Singapore

A project we started a long time ago with Michael Pieler, Philippe Schwaller (he/him) and many others is now on Arxiv: The ChemPile - a massive dataset for training chemical foundation models.

Title grew longer, the benchmark shrank: ChemBench is now out today in Nature Chemistry. rdcu.be/emNWM chembench.lamalab.org behind the paper: communities.springernature.com/posts/behind-t…

I think that most ML benchmarks might be measuring the wrong things entirely. We've been building evaluation tools for ML models in chemistry/materials science for a while now (ChemBench, MaCBench, MatText, etc.), and honestly? I am worried about what we do as a field.