{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/investigating-the-effect-of-natural-language","title":"Investigating the Effect of Natural Language Explanations on Out-of-Distribution Generalization in Few-shot NLI","arxiv_id":"2110.06223","date":"2021-10-12","proceeding":"EMNLP (insights) 2021 11","authors":["Yangqiaoyu Zhou","Chenhao Tan"],"abstract":"Although neural models have shown strong performance in datasets such as SNLI, they lack the ability to generalize out-of-distribution (OOD). In this work, we formulate a few-shot learning setup and examine the effects of natural language explanations on OOD generalization. We leverage the templates in the HANS dataset and construct templated natural language explanations for each template. Although generated explanations show competitive BLEU scores against groundtruth explanations, they fail to improve prediction performance. We further show that generated explanations often hallucinate information and miss key elements that indicate the label.","url_abs":"https://arxiv.org/abs/2110.06223v1","url_pdf":"https://arxiv.org/pdf/2110.06223v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"investigating-the-effect-of-natural-language","repo_url":"https://github.com/chicagohai/hans-explanations","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":null}],"tasks":[{"task_slug":"few-shot-learning","task_name":"Few-Shot Learning"},{"task_slug":"few-shot-nli","task_name":"Few-Shot NLI"},{"task_slug":"out-of-distribution-generalization","task_name":"Out-of-Distribution Generalization"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2110.06223","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2110.06223"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/chicagohai/hans-explanations","reach":null}],"summary":{"ran_violates":1,"ran_draft_wrong":1,"ran_fixture":1},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"fc553819527c9ae2","entry":"contains_dup","repo":"chicagohai/hans-explanations","repo_kind":"official","path":"generate_examples.py","file_url":"https://github.com/chicagohai/hans-explanations/blob/HEAD/generate_examples.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fc553819527c9ae2"}},{"code_sha256_prefix":"c470feec60489b60","entry":"parse_variable","repo":"chicagohai/hans-explanations","repo_kind":"official","path":"generate_examples.py","file_url":"https://github.com/chicagohai/hans-explanations/blob/HEAD/generate_examples.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c470feec60489b60"}},{"code_sha256_prefix":"facfaf05a7045ebd","entry":"split_in_half","repo":"chicagohai/hans-explanations","repo_kind":"official","path":"generate_examples.py","file_url":"https://github.com/chicagohai/hans-explanations/blob/HEAD/generate_examples.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"facfaf05a7045ebd"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}