Large Language Models have proven highly successful at modelling a variety of tasks. However, this comes at a steep computational cost that hinders wider industrial uptake. In this pa005 per, we present MWT: a Multi-Word Tokenizer that goes beyond word boundaries by representing frequent multi-word expressions as single tokens. MWTs produce a more compact and efficient tokenization that yields two benefits: (1) Increase in performance due to a greater coverage of input data given a fixed sequence length and budget; (2) Faster and lighter inference due to the ability to reduce the sequence length with negligible drops in performance. Our results show that MWT is more robust across shorter sequence lengths, thus allowing for major speedups via early sequence truncation.

Gee, L., Rigutini, L., Ernandes, M., Zugarini, A. (2023). Multi-word Tokenization for Sequence Compression. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: Industry Track (pp.612-621). Association for Computational Linguistics [10.18653/v1/2023.emnlp-industry.58].

Multi-word Tokenization for Sequence Compression

Rigutini, Leonardo
Supervision
;
Zugarini, Andrea
Conceptualization
2023-01-01

Abstract

Large Language Models have proven highly successful at modelling a variety of tasks. However, this comes at a steep computational cost that hinders wider industrial uptake. In this pa005 per, we present MWT: a Multi-Word Tokenizer that goes beyond word boundaries by representing frequent multi-word expressions as single tokens. MWTs produce a more compact and efficient tokenization that yields two benefits: (1) Increase in performance due to a greater coverage of input data given a fixed sequence length and budget; (2) Faster and lighter inference due to the ability to reduce the sequence length with negligible drops in performance. Our results show that MWT is more robust across shorter sequence lengths, thus allowing for major speedups via early sequence truncation.
2023
Gee, L., Rigutini, L., Ernandes, M., Zugarini, A. (2023). Multi-word Tokenization for Sequence Compression. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: Industry Track (pp.612-621). Association for Computational Linguistics [10.18653/v1/2023.emnlp-industry.58].
File in questo prodotto:
File Dimensione Formato  
2023.emnlp-industry.58.pdf

accesso aperto

Tipologia: PDF editoriale
Licenza: Creative commons
Dimensione 344 kB
Formato Adobe PDF
344 kB Adobe PDF Visualizza/Apri

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/11365/1255435