{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/an-expanded-massive-multilingual-dataset-for","title":"An Expanded Massive Multilingual Dataset for High-Performance Language Technologies","arxiv_id":"2503.10267","date":"2025-03-13","proceeding":null,"authors":["Laurie Burchell","Ona de Gibert","Nikolay Arefyev","Mikko Aulamo","Marta Bañón","and Pinzhen Chen","Mariia Fedorova","Liane Guillou","Barry Haddow","Jan Hajič","and Jindřich Helcl","Erik Henriksson","Mateusz Klimaszewski","Ville Komulainen","and Andrey Kutuzov","Joona Kytöniemi","Veronika Laippala","Petter Mæhlum","and Bhavitvya Malik","Farrokh Mehryary","Vladislav Mikhailov","Nikita Moghe","and Amanda Myntti","Dayyán O'Brien","Stephan Oepen","Proyag Pal","Jousia Piha","and Sampo Pyysalo","Gema Ramírez-Sánchez","David Samuel","Pavel Stepachev","and Jörg Tiedemann","Dušan Variš","Tereza Vojtěchová","Jaume Zaragoza-Bernabeu"],"abstract":"Training state-of-the-art large language models requires vast amounts of clean and diverse textual data. However, building suitable multilingual datasets remains a challenge. In this work, we present HPLT v2, a collection of high-quality multilingual monolingual and parallel corpora. The monolingual portion of the data contains 8T tokens covering 193 languages, while the parallel data contains 380M sentence pairs covering 51 languages. We document the entire data pipeline and release the code to reproduce it. We provide extensive analysis of the quality and characteristics of our data. Finally, we evaluate the performance of language models and machine translation systems trained on HPLT v2, demonstrating its value.","url_abs":"https://arxiv.org/abs/2503.10267v1","url_pdf":"https://arxiv.org/pdf/2503.10267v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"an-expanded-massive-multilingual-dataset-for","repo_url":"https://github.com/hplt-project/hplt-textpipes","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":null}],"tasks":[{"task_slug":"machine-translation","task_name":"Machine Translation"},{"task_slug":"sentence","task_name":"Sentence"}],"methods":[],"datasets_introduced":[{"slug":"hplt-v2","name":"HPLT v2","full_name":""}],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2503.10267","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}