{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/what-to-pre-train-on-efficient-intermediate","title":"What to Pre-Train on? Efficient Intermediate Task Selection","arxiv_id":"2104.08247","date":"2021-04-16","proceeding":"EMNLP 2021 11","authors":["Clifton Poth","Jonas Pfeiffer","Andreas Rücklé","Iryna Gurevych"],"abstract":"Intermediate task fine-tuning has been shown to culminate in large transfer gains across many NLP tasks. With an abundance of candidate datasets as well as pre-trained language models, it has become infeasible to run the cross-product of all combinations to find the best transfer setting. In this work we first establish that similar sequential fine-tuning gains can be achieved in adapter settings, and subsequently consolidate previously proposed methods that efficiently identify beneficial tasks for intermediate transfer learning. We experiment with a diverse set of 42 intermediate and 11 target English classification, multiple choice, question answering, and sequence tagging tasks. Our results show that efficient embedding based methods that rely solely on the respective datasets outperform computational expensive few-shot fine-tuning approaches. Our best methods achieve an average Regret@3 of less than 1% across all target tasks, demonstrating that we are able to efficiently identify the best datasets for intermediate training.","url_abs":"https://arxiv.org/abs/2104.08247v2","url_pdf":"https://arxiv.org/pdf/2104.08247v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"what-to-pre-train-on-efficient-intermediate","repo_url":"https://github.com/adapter-hub/efficient-task-transfer","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"multiple-choice","task_name":"Multiple-choice"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"transfer-learning","task_name":"Transfer Learning"}],"methods":[{"method_slug":"adapter","method_name":"Adapter"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2104.08247","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2104.08247"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/adapter-hub/efficient-task-transfer","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":6},"by_repo_kind":{"official":{"samples":6,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"0d17aa3f96aa84ea","entry":"compute_Fisher","repo":"adapter-hub/efficient-task-transfer","repo_kind":"official","path":"task_selection/taskemb.py","file_url":"https://github.com/adapter-hub/efficient-task-transfer/blob/HEAD/task_selection/taskemb.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0d17aa3f96aa84ea"}},{"code_sha256_prefix":"c7c8798f0a8242ad","entry":"compute_Fisher_no_labels_cr","repo":"adapter-hub/efficient-task-transfer","repo_kind":"official","path":"task_selection/taskemb.py","file_url":"https://github.com/adapter-hub/efficient-task-transfer/blob/HEAD/task_selection/taskemb.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c7c8798f0a8242ad"}},{"code_sha256_prefix":"d7de9f5a86551bf4","entry":"compute_Fisher_no_labels_qa","repo":"adapter-hub/efficient-task-transfer","repo_kind":"official","path":"task_selection/taskemb.py","file_url":"https://github.com/adapter-hub/efficient-task-transfer/blob/HEAD/task_selection/taskemb.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d7de9f5a86551bf4"}},{"code_sha256_prefix":"bbfc47d976ee7225","entry":"create_position_ids_from_input_ids","repo":"adapter-hub/efficient-task-transfer","repo_kind":"official","path":"task_selection/modeling/modeling_roberta.py","file_url":"https://github.com/adapter-hub/efficient-task-transfer/blob/HEAD/task_selection/modeling/modeling_roberta.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bbfc47d976ee7225"}},{"code_sha256_prefix":"5daa4a28d9949694","entry":"get_full_name","repo":"adapter-hub/efficient-task-transfer","repo_kind":"official","path":"task_selection/utils.py","file_url":"https://github.com/adapter-hub/efficient-task-transfer/blob/HEAD/task_selection/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5daa4a28d9949694"}},{"code_sha256_prefix":"26be70dca3249c0b","entry":"load_tf_weights_in_bert","repo":"adapter-hub/efficient-task-transfer","repo_kind":"official","path":"task_selection/modeling/modeling_bert.py","file_url":"https://github.com/adapter-hub/efficient-task-transfer/blob/HEAD/task_selection/modeling/modeling_bert.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"26be70dca3249c0b"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}