{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/crosslingual-generalization-through-multitask","title":"Crosslingual Generalization through Multitask Finetuning","arxiv_id":"2211.01786","date":"2022-11-03","proceeding":null,"authors":["Niklas Muennighoff","Thomas Wang","Lintang Sutawika","Adam Roberts","Stella Biderman","Teven Le Scao","M Saiful Bari","Sheng Shen","Zheng-Xin Yong","Hailey Schoelkopf","Xiangru Tang","Dragomir Radev","Alham Fikri Aji","Khalid Almubarak","Samuel Albanie","Zaid Alyafeai","Albert Webson","Edward Raff","Colin Raffel"],"abstract":"Multitask prompted finetuning (MTF) has been shown to help large language models generalize to new tasks in a zero-shot setting, but so far explorations of MTF have focused on English data and models. We apply MTF to the pretrained multilingual BLOOM and mT5 model families to produce finetuned variants called BLOOMZ and mT0. We find finetuning large multilingual language models on English tasks with English prompts allows for task generalization to non-English languages that appear only in the pretraining corpus. Finetuning on multilingual tasks with English prompts further improves performance on English and non-English tasks leading to various state-of-the-art zero-shot results. We also investigate finetuning on multilingual tasks with prompts that have been machine-translated from English to match the language of each dataset. We find training on these machine-translated prompts leads to better performance on human-written prompts in the respective languages. Surprisingly, we find models are capable of zero-shot generalization to tasks in languages they have never intentionally seen. We conjecture that the models are learning higher-level capabilities that are both task- and language-agnostic. In addition, we introduce xP3, a composite of supervised datasets in 46 languages with English and machine-translated prompts. Our code, datasets and models are freely available at https://github.com/bigscience-workshop/xmtf.","url_abs":"https://arxiv.org/abs/2211.01786v2","url_pdf":"https://arxiv.org/pdf/2211.01786v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"crosslingual-generalization-through-multitask","repo_url":"https://github.com/bigscience-workshop/xmtf","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"coreference-resolution","task_name":"Coreference Resolution"},{"task_slug":"cross-lingual-transfer","task_name":"Cross-Lingual Transfer"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"sentence-completion","task_name":"Sentence Completion"},{"task_slug":"zero-shot-learning","task_name":"Zero-Shot Learning"},{"task_slug":"zero-shot-generalization","task_name":"Zero-shot Generalization"}],"methods":[{"method_slug":"adafactor","method_name":"Adafactor"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bloomz","method_name":"BLOOMZ"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"glu","method_name":"Gated Linear Unit"},{"method_slug":"inverse-square-root-schedule","method_name":"Inverse Square Root Schedule"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"sentencepiece","method_name":"SentencePiece"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"t5","method_name":"T5"},{"method_slug":"mt0","method_name":"mT0"},{"method_slug":"mt5","method_name":"mT5"}],"datasets_introduced":[{"slug":"xp3","name":"xP3","full_name":""}],"methods_introduced":[{"slug":"bloomz","name":"BLOOMZ","full_name":"BLOOMZ"}],"results":[{"leaderboard":"/sota/coreference-resolution-on-xwinograd-en","task":"Coreference Resolution","dataset":"XWinograd EN","model":"mT0-13B","rank_in_archive_order":1,"of":2,"metrics":{"Accuracy":"81.29"},"uses_additional_data":false},{"leaderboard":"/sota/coreference-resolution-on-xwinograd-en","task":"Coreference Resolution","dataset":"XWinograd EN","model":"BLOOMZ","rank_in_archive_order":2,"of":2,"metrics":{"Accuracy":"69.08"},"uses_additional_data":false},{"leaderboard":"/sota/coreference-resolution-on-xwinograd-fr","task":"Coreference Resolution","dataset":"XWinograd FR","model":"mT0-13B","rank_in_archive_order":1,"of":2,"metrics":{"Accuracy":"78.31"},"uses_additional_data":false},{"leaderboard":"/sota/coreference-resolution-on-xwinograd-fr","task":"Coreference Resolution","dataset":"XWinograd FR","model":"BLOOMZ","rank_in_archive_order":2,"of":2,"metrics":{"Accuracy":"68.67"},"uses_additional_data":false},{"leaderboard":"/sota/cross-lingual-transfer-on-xcopa","task":"Cross-Lingual Transfer","dataset":"XCOPA","model":"mT0-13B","rank_in_archive_order":2,"of":6,"metrics":{"Accuracy":"84.45"},"uses_additional_data":false},{"leaderboard":"/sota/cross-lingual-transfer-on-xcopa","task":"Cross-Lingual Transfer","dataset":"XCOPA","model":"BLOOMZ","rank_in_archive_order":4,"of":6,"metrics":{"Accuracy":"75.5"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-storycloze","task":"Question Answering","dataset":"StoryCloze","model":"BLOOMZ","rank_in_archive_order":1,"of":23,"metrics":{"Accuracy":"96.3"},"uses_additional_data":false}],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2211.01786","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2211.01786"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/bigscience-workshop/xmtf","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":1,"unverified":3},"by_repo_kind":{"official":{"samples":4,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"0b1cb631e6e315b0","entry":"removeHyphen","repo":"bigscience-workshop/xmtf","repo_kind":"official","path":"create_xp3x.py","file_url":"https://github.com/bigscience-workshop/xmtf/blob/HEAD/create_xp3x.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0b1cb631e6e315b0"}},{"code_sha256_prefix":"3cea2e48e72663a4","entry":"apply_template","repo":"bigscience-workshop/xmtf","repo_kind":"official","path":"create_xp3x.py","file_url":"https://github.com/bigscience-workshop/xmtf/blob/HEAD/create_xp3x.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3cea2e48e72663a4"}},{"code_sha256_prefix":"b548f6ca8e44e7dc","entry":"compute_lev","repo":"bigscience-workshop/xmtf","repo_kind":"official","path":"plotstables/levenshtein.py","file_url":"https://github.com/bigscience-workshop/xmtf/blob/HEAD/plotstables/levenshtein.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b548f6ca8e44e7dc"}},{"code_sha256_prefix":"e3b3dd5f405fc865","entry":"count_comments","repo":"bigscience-workshop/xmtf","repo_kind":"official","path":"plotstables/countcode.py","file_url":"https://github.com/bigscience-workshop/xmtf/blob/HEAD/plotstables/countcode.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e3b3dd5f405fc865"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}