{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/mind-the-trade-off-debiasing-nlu-models","title":"Mind the Trade-off: Debiasing NLU Models without Degrading the In-distribution Performance","arxiv_id":"2005.00315","date":"2020-05-01","proceeding":"ACL 2020 6","authors":["Prasetya Ajie Utama","Nafise Sadat Moosavi","Iryna Gurevych"],"abstract":"Models for natural language understanding (NLU) tasks often rely on the idiosyncratic biases of the dataset, which make them brittle against test cases outside the training distribution. Recently, several proposed debiasing methods are shown to be very effective in improving out-of-distribution performance. However, their improvements come at the expense of performance drop when models are evaluated on the in-distribution data, which contain examples with higher diversity. This seemingly inevitable trade-off may not tell us much about the changes in the reasoning and understanding capabilities of the resulting models on broader types of examples beyond the small subset represented in the out-of-distribution data. In this paper, we address this trade-off by introducing a novel debiasing method, called confidence regularization, which discourage models from exploiting biases while enabling them to receive enough incentive to learn from all the training examples. We evaluate our method on three NLU tasks and show that, in contrast to its predecessors, it improves the performance on out-of-distribution datasets (e.g., 7pp gain on HANS dataset) while maintaining the original in-distribution accuracy.","url_abs":"https://arxiv.org/abs/2005.00315v1","url_pdf":"https://arxiv.org/pdf/2005.00315v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"mind-the-trade-off-debiasing-nlu-models","repo_url":"https://github.com/UKPLab/acl2020-confidence-regularization","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"diversity","task_name":"Diversity"},{"task_slug":"natural-language-understanding","task_name":"Natural Language Understanding"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2005.00315","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2005.00315"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/UKPLab/acl2020-confidence-regularization","reach":null}],"summary":{"ran":1,"unverified":1},"by_repo_kind":{"official":{"samples":2,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"83e61e2ce0740427","entry":"ThetaSmoothedDistillLoss","repo":"UKPLab/acl2020-confidence-regularization","repo_kind":"official","path":"src/clf_distill_loss_functions.py","file_url":"https://github.com/UKPLab/acl2020-confidence-regularization/blob/HEAD/src/clf_distill_loss_functions.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"83e61e2ce0740427"}},{"code_sha256_prefix":"0f47a012c73ff153","entry":"ClfDistillLossFunction","repo":"UKPLab/acl2020-confidence-regularization","repo_kind":"official","path":"src/clf_distill_loss_functions.py","file_url":"https://github.com/UKPLab/acl2020-confidence-regularization/blob/HEAD/src/clf_distill_loss_functions.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0f47a012c73ff153"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}