{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/english-intermediate-task-training-improves","title":"English Intermediate-Task Training Improves Zero-Shot Cross-Lingual Transfer Too","arxiv_id":"2005.13013","date":"2020-05-26","proceeding":"Asian Chapter of the Association for Computational Linguistics 2020","authors":["Jason Phang","Iacer Calixto","Phu Mon Htut","Yada Pruksachatkun","Haokun Liu","Clara Vania","Katharina Kann","Samuel R. Bowman"],"abstract":"Intermediate-task training---fine-tuning a pretrained model on an intermediate task before fine-tuning again on the target task---often improves model performance substantially on language understanding tasks in monolingual English settings. We investigate whether English intermediate-task training is still helpful on non-English target tasks. Using nine intermediate language-understanding tasks, we evaluate intermediate-task transfer in a zero-shot cross-lingual setting on the XTREME benchmark. We see large improvements from intermediate training on the BUCC and Tatoeba sentence retrieval tasks and moderate improvements on question-answering target tasks. MNLI, SQuAD and HellaSwag achieve the best overall results as intermediate tasks, while multi-task intermediate offers small additional improvements. Using our best intermediate-task models for each target task, we obtain a 5.4 point improvement over XLM-R Large on the XTREME benchmark, setting the state of the art as of June 2020. We also investigate continuing multilingual MLM during intermediate-task training and using machine-translated intermediate-task data, but neither consistently outperforms simply performing English intermediate-task training.","url_abs":"https://arxiv.org/abs/2005.13013v2","url_pdf":"https://arxiv.org/pdf/2005.13013v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":"cross-lingual-transfer","task_name":"Cross-Lingual Transfer"},{"task_slug":"hellaswag","task_name":"HellaSwag"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"retrieval","task_name":"Retrieval"},{"task_slug":"sentence","task_name":"Sentence"},{"task_slug":"sentence-retrieval","task_name":"Sentence Retrieval"},{"task_slug":"xlm-r","task_name":"XLM-R"},{"task_slug":"zero-shot-cross-lingual-transfer","task_name":"Zero-Shot Cross-Lingual Transfer"}],"methods":[{"method_slug":"xlm-r","method_name":"XLM-R"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/zero-shot-cross-lingual-transfer-on-xtreme","task":"Zero-Shot Cross-Lingual Transfer","dataset":"XTREME","model":"X-STILTs","rank_in_archive_order":20,"of":25,"metrics":{"Avg":"73.5","Question Answering":"67.2","Sentence Retrieval":"76.5","Sentence-pair Classification":"83.9","Structured Prediction":"69.4"},"uses_additional_data":false}],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2005.13013","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}