{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/the-change-that-matters-in-discourse-parsing-1","title":"The Change that Matters in Discourse Parsing: Estimating the Impact of Domain Shift on Parser Error","arxiv_id":"2203.11317","date":"2022-03-21","proceeding":"Findings (ACL) 2022 5","authors":["Katherine Atwell","Anthony Sicilia","Seong Jae Hwang","Malihe Alikhani"],"abstract":"Discourse analysis allows us to attain inferences of a text document that extend beyond the sentence-level. The current performance of discourse models is very low on texts outside of the training distribution's coverage, diminishing the practical utility of existing models. There is need for a measure that can inform us to what extent our model generalizes from the training to the test sample when these samples may be drawn from distinct distributions. While this can be estimated via distribution shift, we argue that this does not directly correlate with change in the observed error of a classifier (i.e. error-gap). Thus, we propose to use a statistic from the theoretical domain adaptation literature which can be directly tied to error-gap. We study the bias of this statistic as an estimator of error-gap both theoretically and through a large-scale empirical study of over 2400 experiments on 6 discourse datasets from domains including, but not limited to: news, biomedical texts, TED talks, Reddit posts, and fiction. Our results not only motivate our proposal and help us to understand its limitations, but also provide insight on the properties of discourse models and datasets which improve performance in domain adaptation. For instance, we find that non-news datasets are slightly easier to transfer to than news datasets when the training and test sets are very different. Our code and an associated Python package are available to allow practitioners to make more informed model and dataset choices.","url_abs":"https://arxiv.org/abs/2203.11317v1","url_pdf":"https://arxiv.org/pdf/2203.11317v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"the-change-that-matters-in-discourse-parsing-1","repo_url":"https://github.com/anthonysicilia/change-that-matters-acl2022","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"the-change-that-matters-in-discourse-parsing-1","repo_url":"https://github.com/anthonysicilia/equitable-dialogue-acl2023","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"the-change-that-matters-in-discourse-parsing-1","repo_url":"https://github.com/anthonysicilia/leather-aacl2022","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"the-change-that-matters-in-discourse-parsing-1","repo_url":"https://github.com/anthonysicilia/pacbayes-adaptation-uai2022","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"discourse-parsing","task_name":"Discourse Parsing"},{"task_slug":"domain-adaptation","task_name":"Domain Adaptation"},{"task_slug":"sentence","task_name":"Sentence"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2203.11317","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2203.11317"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/anthonysicilia/change-that-matters-ACL2022","reach":null}],"summary":{"ran_draft_wrong":2,"unverified":3},"by_repo_kind":{"official":{"samples":5,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"6b7ddaf8c7bd89e9","entry":"interactions","repo":"anthonysicilia/change-that-matters-ACL2022","repo_kind":"official","path":"lreg.py","file_url":"https://github.com/anthonysicilia/change-that-matters-ACL2022/blob/HEAD/lreg.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6b7ddaf8c7bd89e9"}},{"code_sha256_prefix":"9df0a1645192d34e","entry":"lm","repo":"anthonysicilia/change-that-matters-ACL2022","repo_kind":"official","path":"lreg.py","file_url":"https://github.com/anthonysicilia/change-that-matters-ACL2022/blob/HEAD/lreg.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9df0a1645192d34e"}},{"code_sha256_prefix":"d1454c19359f50f2","entry":"_plot_residuals","repo":"anthonysicilia/change-that-matters-ACL2022","repo_kind":"official","path":"lreg.py","file_url":"https://github.com/anthonysicilia/change-that-matters-ACL2022/blob/HEAD/lreg.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d1454c19359f50f2"}},{"code_sha256_prefix":"f483a3e051fc1dde","entry":"lm_diag","repo":"anthonysicilia/change-that-matters-ACL2022","repo_kind":"official","path":"lreg.py","file_url":"https://github.com/anthonysicilia/change-that-matters-ACL2022/blob/HEAD/lreg.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f483a3e051fc1dde"}},{"code_sha256_prefix":"370ed4b18a3a7d9f","entry":"plot_residuals","repo":"anthonysicilia/change-that-matters-ACL2022","repo_kind":"official","path":"lreg.py","file_url":"https://github.com/anthonysicilia/change-that-matters-ACL2022/blob/HEAD/lreg.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"370ed4b18a3a7d9f"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}