{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/single-dataset-experts-for-multi-dataset","title":"Single-dataset Experts for Multi-dataset Question Answering","arxiv_id":"2109.13880","date":"2021-09-28","proceeding":"EMNLP 2021 11","authors":["Dan Friedman","Ben Dodge","Danqi Chen"],"abstract":"Many datasets have been created for training reading comprehension models, and a natural question is whether we can combine them to build models that (1) perform better on all of the training datasets and (2) generalize and transfer better to new datasets. Prior work has addressed this goal by training one network simultaneously on multiple datasets, which works well on average but is prone to over- or under-fitting different sub-distributions and might transfer worse compared to source models with more overlap with the target dataset. Our approach is to model multi-dataset question answering with a collection of single-dataset experts, by training a collection of lightweight, dataset-specific adapter modules (Houlsby et al., 2019) that share an underlying Transformer model. We find that these Multi-Adapter Dataset Experts (MADE) outperform all our baselines in terms of in-distribution accuracy, and simple methods based on parameter-averaging lead to better zero-shot generalization and few-shot transfer performance, offering a strong and versatile starting point for building new reading comprehension systems.","url_abs":"https://arxiv.org/abs/2109.13880v1","url_pdf":"https://arxiv.org/pdf/2109.13880v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"single-dataset-experts-for-multi-dataset","repo_url":"https://github.com/princeton-nlp/made","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"reading-comprehension","task_name":"Reading Comprehension"},{"task_slug":"zero-shot-generalization","task_name":"Zero-shot Generalization"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"adapter","method_name":"Adapter"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2109.13880","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2109.13880"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/princeton-nlp/made","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/princeton-nlp/MADE","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_violates":1,"ran_honours":1,"ran_draft_wrong":1,"unverified":8},"by_repo_kind":{"official":{"samples":11,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"bf75673211903e14","entry":"exact_match_score","repo":"princeton-nlp/MADE","repo_kind":"official","path":"src/utils/metrics.py","file_url":"https://github.com/princeton-nlp/MADE/blob/HEAD/src/utils/metrics.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bf75673211903e14"}},{"code_sha256_prefix":"a8ef93fef742e557","entry":"f1_score","repo":"princeton-nlp/MADE","repo_kind":"official","path":"src/utils/metrics.py","file_url":"https://github.com/princeton-nlp/MADE/blob/HEAD/src/utils/metrics.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a8ef93fef742e557"}},{"code_sha256_prefix":"e7e75981cb464788","entry":"normalize_answer","repo":"princeton-nlp/MADE","repo_kind":"official","path":"src/utils/metrics.py","file_url":"https://github.com/princeton-nlp/MADE/blob/HEAD/src/utils/metrics.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e7e75981cb464788"}},{"code_sha256_prefix":"f99a8dce39e0242c","entry":"asdict_","repo":"princeton-nlp/MADE","repo_kind":"official","path":"src/models/trainer.py","file_url":"https://github.com/princeton-nlp/MADE/blob/HEAD/src/models/trainer.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f99a8dce39e0242c"}},{"code_sha256_prefix":"164518300be9d42f","entry":"char_span_to_token_span","repo":"princeton-nlp/MADE","repo_kind":"official","path":"src/utils/data_utils.py","file_url":"https://github.com/princeton-nlp/MADE/blob/HEAD/src/utils/data_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"164518300be9d42f"}},{"code_sha256_prefix":"12246540982c0879","entry":"get_handler","repo":"princeton-nlp/MADE","repo_kind":"official","path":"src/utils/logging.py","file_url":"https://github.com/princeton-nlp/MADE/blob/HEAD/src/utils/logging.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"12246540982c0879"}},{"code_sha256_prefix":"13444437d74ae65b","entry":"get_logger","repo":"princeton-nlp/MADE","repo_kind":"official","path":"src/utils/logging.py","file_url":"https://github.com/princeton-nlp/MADE/blob/HEAD/src/utils/logging.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"13444437d74ae65b"}},{"code_sha256_prefix":"266398f337ae2bc4","entry":"get_resume_name","repo":"princeton-nlp/MADE","repo_kind":"official","path":"src/utils/logging.py","file_url":"https://github.com/princeton-nlp/MADE/blob/HEAD/src/utils/logging.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"266398f337ae2bc4"}},{"code_sha256_prefix":"101cd5b769362370","entry":"marginal_ll","repo":"princeton-nlp/MADE","repo_kind":"official","path":"src/utils/model_utils.py","file_url":"https://github.com/princeton-nlp/MADE/blob/HEAD/src/utils/model_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"101cd5b769362370"}},{"code_sha256_prefix":"4526f5f32fe3ec45","entry":"max_ll","repo":"princeton-nlp/MADE","repo_kind":"official","path":"src/utils/model_utils.py","file_url":"https://github.com/princeton-nlp/MADE/blob/HEAD/src/utils/model_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4526f5f32fe3ec45"}},{"code_sha256_prefix":"c5fd58d7377e99ce","entry":"to_device","repo":"princeton-nlp/MADE","repo_kind":"official","path":"src/utils/model_utils.py","file_url":"https://github.com/princeton-nlp/MADE/blob/HEAD/src/utils/model_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c5fd58d7377e99ce"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}