{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/polylm-an-open-source-polyglot-large-language","title":"PolyLM: An Open Source Polyglot Large Language Model","arxiv_id":"2307.06018","date":"2023-07-12","proceeding":null,"authors":["Xiangpeng Wei","Haoran Wei","Huan Lin","TianHao Li","Pei Zhang","Xingzhang Ren","Mei Li","Yu Wan","Zhiwei Cao","Binbin Xie","Tianxiang Hu","Shangjie Li","Binyuan Hui","Bowen Yu","Dayiheng Liu","Baosong Yang","Fei Huang","Jun Xie"],"abstract":"Large language models (LLMs) demonstrate remarkable ability to comprehend, reason, and generate following nature language instructions. However, the development of LLMs has been primarily focused on high-resource languages, such as English, thereby limiting their applicability and research in other languages. Consequently, we present PolyLM, a multilingual LLM trained on 640 billion (B) tokens, avaliable in two model sizes: 1.7B and 13B. To enhance its multilingual capabilities, we 1) integrate bilingual data into training data; and 2) adopt a curriculum learning strategy that increases the proportion of non-English data from 30% in the first stage to 60% in the final stage during pre-training. Further, we propose a multilingual self-instruct method which automatically generates 132.7K diverse multilingual instructions for model fine-tuning. To assess the model's performance, we collect several existing multilingual tasks, including multilingual understanding, question answering, generation, and translation. Extensive experiments show that PolyLM surpasses other open-source models such as LLaMA and BLOOM on multilingual tasks while maintaining comparable performance in English. Our models, alone with the instruction data and multilingual benchmark, are available at: \\url{https://modelscope.cn/models/damo/nlp_polylm_13b_text_generation}.","url_abs":"https://arxiv.org/abs/2307.06018v1","url_pdf":"https://arxiv.org/pdf/2307.06018v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"polylm-an-open-source-polyglot-large-language","repo_url":"https://github.com/modelscope/modelscope","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"large-language-model","task_name":"Large Language Model"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"model","task_name":"model"}],"methods":[{"method_slug":"bloom","method_name":"BLOOM"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2307.06018","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}