{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/exploring-the-benefits-of-training-expert","title":"Exploring the Benefits of Training Expert Language Models over Instruction Tuning","arxiv_id":"2302.03202","date":"2023-02-07","proceeding":null,"authors":["Joel Jang","Seungone Kim","Seonghyeon Ye","Doyoung Kim","Lajanugen Logeswaran","Moontae Lee","Kyungjae Lee","Minjoon Seo"],"abstract":"Recently, Language Models (LMs) instruction-tuned on multiple tasks, also known as multitask-prompted fine-tuning (MT), have shown the capability to generalize to unseen tasks. Previous work has shown that scaling the number of training tasks is the key component in making stronger MT LMs. In this work, we report an unexpected finding that an expert LM fine-tuned on just a single task can outperform an MT LM trained with 300+ different tasks on 11 different unseen datasets and on 13 datasets of the BIG-bench benchmark by a mean accuracy of 3.20% and 1.29%, respectively. This finding casts doubt on the previously held belief that simply scaling the number of tasks makes stronger MT LMs. Leveraging this finding, we further show that this distributed approach of training a separate expert LM per training task instead of a single MT LM for zero-shot inference possesses many benefits including (1) avoiding negative task transfer that often occurs during instruction tuning, (2) being able to continually learn new tasks without having to re-train on previous tasks to avoid catastrophic forgetting, and (3) showing compositional capabilities when merging individual experts together. The code is available at https://github.com/joeljang/ELM.","url_abs":"https://arxiv.org/abs/2302.03202v2","url_pdf":"https://arxiv.org/pdf/2302.03202v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"exploring-the-benefits-of-training-expert","repo_url":"https://github.com/joeljang/elm","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"exploring-the-benefits-of-training-expert","repo_url":"https://github.com/joeljang/rlphf","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"common-sense-reasoning","task_name":"Common Sense Reasoning"},{"task_slug":"coreference-resolution","task_name":"Coreference Resolution"},{"task_slug":"natural-language-inference","task_name":"Natural Language Inference"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"sentence-completion","task_name":"Sentence Completion"},{"task_slug":"word-sense-disambiguation","task_name":"Word Sense Disambiguation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/common-sense-reasoning-on-winogrande","task":"Common Sense Reasoning","dataset":"WinoGrande","model":"RoE-3B","rank_in_archive_order":48,"of":77,"metrics":{"Accuracy":"61.60"},"uses_additional_data":false},{"leaderboard":"/sota/coreference-resolution-on-winograd-schema","task":"Coreference Resolution","dataset":"Winograd Schema Challenge","model":"RoE-3B","rank_in_archive_order":52,"of":82,"metrics":{"Accuracy":"62.21"},"uses_additional_data":false},{"leaderboard":"/sota/natural-language-inference-on-anli-test","task":"Natural Language Inference","dataset":"ANLI test","model":"RoE-3B","rank_in_archive_order":15,"of":25,"metrics":{"A1":"35.49","A2":"34.64","A3":"31.22"},"uses_additional_data":false},{"leaderboard":"/sota/natural-language-inference-on-rte","task":"Natural Language Inference","dataset":"RTE","model":"RoE-3B","rank_in_archive_order":66,"of":90,"metrics":{"Accuracy":"64.01"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-copa","task":"Question Answering","dataset":"COPA","model":"RoE-3B","rank_in_archive_order":37,"of":60,"metrics":{"Accuracy":"79.25"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-storycloze","task":"Question Answering","dataset":"StoryCloze","model":"RoE-3B","rank_in_archive_order":9,"of":23,"metrics":{"Accuracy":"86.33"},"uses_additional_data":false},{"leaderboard":"/sota/sentence-completion-on-hellaswag","task":"Sentence Completion","dataset":"HellaSwag","model":"RoE-3B","rank_in_archive_order":81,"of":89,"metrics":{"Accuracy":"34.6"},"uses_additional_data":false},{"leaderboard":"/sota/word-sense-disambiguation-on-words-in-context","task":"Word Sense Disambiguation","dataset":"Words in Context","model":"RoE-3B","rank_in_archive_order":25,"of":37,"metrics":{"Accuracy":"52.97"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2302.03202","atlas_url":"https://app.syntology.ai/?focus=2302.03202","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2302.03202"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/joeljang/elm","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/joeljang/rlphf","reach":{"status":"ok"}}],"summary":{"ran_draft_wrong":3},"by_repo_kind":{"official":{"samples":2,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"9aa2aaf773117b75","entry":"clease_jinja","repo":"joeljang/elm","repo_kind":"official","path":"retrieval/run_retrieval.py","file_url":"https://github.com/joeljang/elm/blob/HEAD/retrieval/run_retrieval.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9aa2aaf773117b75"}},{"code_sha256_prefix":"d77e7ade95e0c350","entry":"euclidean_distance","repo":"joeljang/elm","repo_kind":"official","path":"retrieval/run_retrieval.py","file_url":"https://github.com/joeljang/elm/blob/HEAD/retrieval/run_retrieval.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d77e7ade95e0c350"}},{"code_sha256_prefix":"44e48dfbdecee3da","entry":"run_command","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"44e48dfbdecee3da"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}