Maite Melero
@maitemelero1
ID: 809736524887683072
16-12-2016 12:26:46
636 Tweet
186 Followers
232 Following
[2/7] Along with the paper we release PLUME a family of 3 2B #LLM based on the Gemma architecture. Each model uses a different vocabulary size, from 32k up to 256k tokens. PLUME 32k: huggingface.co/projecte-aina/… PLUME 128k: huggingface.co/projecte-aina/… PLUME 256k: huggingface.co/projecte-aina/…
![[3/7] Our results show that these models can perform comparably to previous Encoder-Decoder methods and that larger vocabularies lead to better performance on both supervised and zero-shot translation directions.](https://pbs.twimg.com/media/GQBX4tEW8AAbCTO.png "Carlos Escolano (@carlosep93) on Twitter photo [3/7] Our results show that these models can perform comparably to previous Encoder-Decoder methods and that larger vocabularies lead to better performance on both supervised and zero-shot translation directions.")
![[4/7] Further analysis shows that different layers specialize in different parts of the prompt. Two clear patterns we observe are the presence of sink heads that attend to the <BOS> token, and a small amount of attention to the source language tag.](https://pbs.twimg.com/media/GQBZ37zWMAAFLMl.jpg "Carlos Escolano (@carlosep93) on Twitter photo [4/7] Further analysis shows that different layers specialize in different parts of the prompt. Two clear patterns we observe are the presence of sink heads that attend to the <BOS> token, and a small amount of attention to the source language tag.")
![[5/7] Given the previous findings we masked the heads with less attention coverage. Results show that more than 47% of model heads can be removed without losing more than 2 BLEU points. Being the 256k model the more resilient one, with 64,7% masked heads.](https://pbs.twimg.com/media/GQBbWWQWwAE65Z0.jpg "Carlos Escolano (@carlosep93) on Twitter photo [5/7] Given the previous findings we masked the heads with less attention coverage. Results show that more than 47% of model heads can be removed without losing more than 2 BLEU points. Being the 256k model the more resilient one, with 64,7% masked heads.")
![[6/7] Finally, we study how the cross-lingual space is learned through the model layers. We observe that larger vocabulary sizes show smaller distances between languages, at the early and middle layers.](https://pbs.twimg.com/media/GQBcA6eWsAAb87z.jpg "Carlos Escolano (@carlosep93) on Twitter photo [6/7] Finally, we study how the cross-lingual space is learned through the model layers. We observe that larger vocabulary sizes show smaller distances between languages, at the early and middle layers.")
[7/7] This work has been conducted at BSC-CNS thanks to funding by Aina and Proyecto Ilenia. Also, thank my co-authors Javier García Gilabert (Javier García Gilabert), Aleix Sant Savall, Francesca De Luca Fornaciari, Audrey Mash, Xixian Liao, Maite Melero (Maite Melero )
L'Alternativa 2024 - ALTO EL FUEGO alternativa.cccb.org/2024/es/fest/c… via L'Alternativa Fest
