{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/learning-to-model-and-ignore-dataset-bias","title":"Learning to Model and Ignore Dataset Bias with Mixed Capacity Ensembles","arxiv_id":"2011.03856","date":"2020-11-07","proceeding":"Findings of the Association for Computational Linguistics 2020","authors":["Christopher Clark","Mark Yatskar","Luke Zettlemoyer"],"abstract":"Many datasets have been shown to contain incidental correlations created by idiosyncrasies in the data collection process. For example, sentence entailment datasets can have spurious word-class correlations if nearly all contradiction sentences contain the word \"not\", and image recognition datasets can have tell-tale object-background correlations if dogs are always indoors. In this paper, we propose a method that can automatically detect and ignore these kinds of dataset-specific patterns, which we call dataset biases. Our method trains a lower capacity model in an ensemble with a higher capacity model. During training, the lower capacity model learns to capture relatively shallow correlations, which we hypothesize are likely to reflect dataset bias. This frees the higher capacity model to focus on patterns that should generalize better. We ensure the models learn non-overlapping approaches by introducing a novel method to make them conditionally independent. Importantly, our approach does not require the bias to be known in advance. We evaluate performance on synthetic datasets, and four datasets built to penalize models that exploit known biases on textual entailment, visual question answering, and image recognition tasks. We show improvement in all settings, including a 10 point gain on the visual question answering dataset.","url_abs":"https://arxiv.org/abs/2011.03856v1","url_pdf":"https://arxiv.org/pdf/2011.03856v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"learning-to-model-and-ignore-dataset-bias","repo_url":"https://github.com/chrisc36/autobias","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"natural-language-inference","task_name":"Natural Language Inference"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"sentence","task_name":"Sentence"},{"task_slug":"visual-question-answering-1","task_name":"Visual Question Answering"},{"task_slug":"visual-question-answering","task_name":"Visual Question Answering (VQA)"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2011.03856","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2011.03856"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/chrisc36/autobias","reach":null}],"summary":{"ran":5,"ran_draft_wrong":2,"ran_violates":1,"ran_honours":2,"unverified":8},"by_repo_kind":{"official":{"samples":18,"ran":10,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"66d942a22936224d","entry":"ArgminFunction","repo":"chrisc36/autobias","repo_kind":"official","path":"autobias/modules/classifier_ensembles.py","file_url":"https://github.com/chrisc36/autobias/blob/HEAD/autobias/modules/classifier_ensembles.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"66d942a22936224d"}},{"code_sha256_prefix":"1517452197d65b98","entry":"ClassifierEnsembleOutput","repo":"chrisc36/autobias","repo_kind":"official","path":"autobias/modules/classifier_ensembles.py","file_url":"https://github.com/chrisc36/autobias/blob/HEAD/autobias/modules/classifier_ensembles.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1517452197d65b98"}},{"code_sha256_prefix":"4407441e018c9820","entry":"Configuration","repo":"chrisc36/autobias","repo_kind":"official","path":"autobias/modules/classifier_ensembles.py","file_url":"https://github.com/chrisc36/autobias/blob/HEAD/autobias/modules/classifier_ensembles.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4407441e018c9820"}},{"code_sha256_prefix":"1788407f38a54f42","entry":"Layer","repo":"chrisc36/autobias","repo_kind":"official","path":"autobias/modules/classifier_ensembles.py","file_url":"https://github.com/chrisc36/autobias/blob/HEAD/autobias/modules/classifier_ensembles.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1788407f38a54f42"}},{"code_sha256_prefix":"329d6081e2b7090a","entry":"TrainOutput","repo":"chrisc36/autobias","repo_kind":"official","path":"autobias/modules/classifier_ensembles.py","file_url":"https://github.com/chrisc36/autobias/blob/HEAD/autobias/modules/classifier_ensembles.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"329d6081e2b7090a"}},{"code_sha256_prefix":"c24ddaada1a1c755","entry":"_get_param_names","repo":"chrisc36/autobias","repo_kind":"official","path":"autobias/modules/classifier_ensembles.py","file_url":"https://github.com/chrisc36/autobias/blob/HEAD/autobias/modules/classifier_ensembles.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c24ddaada1a1c755"}},{"code_sha256_prefix":"01b501ef47ff7315","entry":"_to_py","repo":"chrisc36/autobias","repo_kind":"official","path":"autobias/modules/classifier_ensembles.py","file_url":"https://github.com/chrisc36/autobias/blob/HEAD/autobias/modules/classifier_ensembles.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"01b501ef47ff7315"}},{"code_sha256_prefix":"3126bcabe72c01de","entry":"compute_class_prior","repo":"chrisc36/autobias","repo_kind":"official","path":"autobias/modules/classifier_ensembles.py","file_url":"https://github.com/chrisc36/autobias/blob/HEAD/autobias/modules/classifier_ensembles.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3126bcabe72c01de"}},{"code_sha256_prefix":"fc370fba23e6b428","entry":"cross_entropy","repo":"chrisc36/autobias","repo_kind":"official","path":"autobias/modules/classifier_ensembles.py","file_url":"https://github.com/chrisc36/autobias/blob/HEAD/autobias/modules/classifier_ensembles.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fc370fba23e6b428"}},{"code_sha256_prefix":"94de706a508f6295","entry":"weighted_mean","repo":"chrisc36/autobias","repo_kind":"official","path":"autobias/modules/classifier_ensembles.py","file_url":"https://github.com/chrisc36/autobias/blob/HEAD/autobias/modules/classifier_ensembles.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"94de706a508f6295"}},{"code_sha256_prefix":"1965ee4e5a5c3bab","entry":"ArgminTransformFunction","repo":"chrisc36/autobias","repo_kind":"official","path":"autobias/modules/classifier_ensembles.py","file_url":"https://github.com/chrisc36/autobias/blob/HEAD/autobias/modules/classifier_ensembles.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1965ee4e5a5c3bab"}},{"code_sha256_prefix":"881087663bfb8da6","entry":"ClfArgminEnsemble","repo":"chrisc36/autobias","repo_kind":"official","path":"autobias/modules/classifier_ensembles.py","file_url":"https://github.com/chrisc36/autobias/blob/HEAD/autobias/modules/classifier_ensembles.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"881087663bfb8da6"}},{"code_sha256_prefix":"9dccbb193bda0913","entry":"ClfHead","repo":"chrisc36/autobias","repo_kind":"official","path":"autobias/modules/classifier_ensembles.py","file_url":"https://github.com/chrisc36/autobias/blob/HEAD/autobias/modules/classifier_ensembles.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9dccbb193bda0913"}},{"code_sha256_prefix":"4a2c1d211e8c974d","entry":"Configured","repo":"chrisc36/autobias","repo_kind":"official","path":"autobias/modules/classifier_ensembles.py","file_url":"https://github.com/chrisc36/autobias/blob/HEAD/autobias/modules/classifier_ensembles.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4a2c1d211e8c974d"}},{"code_sha256_prefix":"8b2698ce36b58c25","entry":"Predictor","repo":"chrisc36/autobias","repo_kind":"official","path":"autobias/modules/classifier_ensembles.py","file_url":"https://github.com/chrisc36/autobias/blob/HEAD/autobias/modules/classifier_ensembles.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8b2698ce36b58c25"}},{"code_sha256_prefix":"d9dc3b79245d226b","entry":"_ConfiguredJSONEncoder","repo":"chrisc36/autobias","repo_kind":"official","path":"autobias/modules/classifier_ensembles.py","file_url":"https://github.com/chrisc36/autobias/blob/HEAD/autobias/modules/classifier_ensembles.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d9dc3b79245d226b"}},{"code_sha256_prefix":"44eeb9ee4b01368a","entry":"config_to_json","repo":"chrisc36/autobias","repo_kind":"official","path":"autobias/modules/classifier_ensembles.py","file_url":"https://github.com/chrisc36/autobias/blob/HEAD/autobias/modules/classifier_ensembles.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"44eeb9ee4b01368a"}},{"code_sha256_prefix":"be373e2a8ecbf2c4","entry":"get_configuration","repo":"chrisc36/autobias","repo_kind":"official","path":"autobias/modules/classifier_ensembles.py","file_url":"https://github.com/chrisc36/autobias/blob/HEAD/autobias/modules/classifier_ensembles.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"be373e2a8ecbf2c4"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}